devbox
bf789
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

文本相似度和文本匹配模型归纳总结（附keras代码）_文本相似性分析

作者：bf789 | 2024-01-28 23:43:11

赞

踩

文本相似性分析

1.文本相似度，文本匹配模型归纳总结

DSSM详解

ESIM详解

ABCNN详解

BiMPM详解

DIIN详解

DRCN详解

    https://blog.csdn.net/u012526436/article/details/90179466

2. 短文本相似度计算方法

https://blog.csdn.net/baidu_26550817/article/details/80171532
最长公共子序列
编辑距离
相同单词个数/序列长度
word2vec+余弦相似度
Sentence2Vector 
https://blog.csdn.net/qjzcy/article/details/51882959?spm=0.0.0.0.zFx7Qk
DSSM(deep structured semantic models)(BOW/CNN/RNN) 
https://www.cnblogs.com/qniguoym/p/7772561.html

lstm+topic 
https://blog.csdn.net/qjzcy/article/details/52269382

百度AI的例子: 
http://ai.baidu.com/tech/nlp/simnet 
http://ai.baidu.com/docs#/NLP-API/c150c35a

3.文本分类


贝叶斯
支持向量
逻辑回归
http://sklearn.apachecn.org/cn/0.19.0/auto_examples/text/document_classification_20newsgroups.html#sphx-glr-auto-examples-text-document-classification-20newsgroups-py
fasttext
bilstm
cnn
rcnn
https://github.com/keras-team/keras/tree/master/examples


4.序列标注


HMM
CRF
LSTM+CRF
seq2seq
seq2seq+attention

5.部分模型keras实现

1. LSTM实现文本相似度：

  1. def get_model(nb_words, EMBEDDING_DIM, embedding_matrix, MAX_SEQUENCE_LENGTH,
  2.                            num_lstm, rate_drop_lstm, rate_drop_dense, num_dense, act):
  3.  
  4.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  5.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  6. 	
  7. 	# embedding
  8.     embedding_layer = Embedding(nb_words,
  9.                                 EMBEDDING_DIM,
  10.                                 weights=[embedding_matrix],
  11.                                 input_length=MAX_SEQUENCE_LENGTH,
  12.                                 trainable=False)
  13.     embedded_sequences_1 = embedding_layer(sequence_1_input)
  14.     embedded_sequences_2 = embedding_layer(sequence_2_input)
  15. 	
  16. 	# lstm
  17.     lstm_layer = LSTM(num_lstm, dropout=rate_drop_lstm, recurrent_dropout=rate_drop_lstm)
  18.     x1 = lstm_layer(embedded_sequences_1)
  19.     y1 = lstm_layer(embedded_sequences_2)
  20.  
  21. 	# classifier
  22.     merged = concatenate([x1, y1])
  23.     merged = Dropout(rate_drop_dense)(merged)
  24.     merged = BatchNormalization()(merged)
  25.     merged = Dense(num_dense, activation=act)(merged)
  26.     merged = Dropout(rate_drop_dense)(merged)
  27.     merged = BatchNormalization()(merged)
  28.     preds = Dense(1, activation='sigmoid')(merged)
  29.  
  30.     model = Model(inputs=[sequence_1_input, sequence_2_input], \
  31.                   outputs=preds)
  32.     model.compile(loss='binary_crossentropy',
  33.                   optimizer='nadam',
  34.                   metrics=['acc'])
  35.     model.summary()
  36.     return model

2. BiLSTM实现文本相似度

  1. def get_model(nb_words, EMBEDDING_DIM, embedding_matrix, MAX_SEQUENCE_LENGTH,
  2.                            num_lstm, rate_drop_lstm, rate_drop_dense, num_dense, act):
  3.     embedding_layer = Embedding(nb_words,
  4.                                 EMBEDDING_DIM,
  5.                                 weights=[embedding_matrix],
  6.                                 input_length=MAX_SEQUENCE_LENGTH,
  7.                                 trainable=False)
  8.     lstm_layer = Bidirectional(LSTM(num_lstm, dropout=rate_drop_lstm, recurrent_dropout=rate_drop_lstm))
  9.  
  10.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  11.     embedded_sequences_1 = embedding_layer(sequence_1_input)
  12.     x1 = lstm_layer(embedded_sequences_1)
  13.  
  14.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  15.     embedded_sequences_2 = embedding_layer(sequence_2_input)
  16.     y1 = lstm_layer(embedded_sequences_2)
  17.  
  18.     merged = concatenate([x1, y1])
  19.     merged = Dropout(rate_drop_dense)(merged)
  20.     merged = BatchNormalization()(merged)
  21.  
  22.     merged = Dense(num_dense, activation=act)(merged)
  23.     merged = Dropout(rate_drop_dense)(merged)
  24.     merged = BatchNormalization()(merged)
  25.     preds = Dense(1, activation='sigmoid')(merged)
  26.  
  27.     model = Model(inputs=[sequence_1_input, sequence_2_input], \
  28.                   outputs=preds)
  29.     model.compile(loss='binary_crossentropy',
  30.                   optimizer='adam',
  31.                   metrics=['acc'])
  32.     model.summary()
  33.     return model

3. ESIM实现文本相似度

  1. def get_model(embedding_matrix_file, MAX_SEQUENCE_LENGTH, num_lstm, rate_drop_dense, num_dense):
  2.  
  3.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  4.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  5.  
  6.     # embedding
  7.     embedding_layer = create_pretrained_embedding(embedding_matrix_file, mask_zero=False)
  8.     bn = BatchNormalization(axis=2)
  9.     embedded_sequences_1 = bn(embedding_layer(sequence_1_input))
  10.     embedded_sequences_2 = bn(embedding_layer(sequence_2_input))
  11.  
  12.     # encode
  13.     encode = Bidirectional(LSTM(num_lstm, return_sequences=True))
  14.     encode_sequences_1 = encode(embedded_sequences_1)
  15.     encode_sequences_2 = encode(embedded_sequences_2)
  16.  
  17.     # attention
  18.     alignd_sequences_1, alignd_sequences_2 = soft_attention_alignment(encode_sequences_1, encode_sequences_2)
  19.  
  20.     # compose
  21.     combined_sequences_1 = Concatenate()(
  22.         [encode_sequences_1, alignd_sequences_2, submult(encode_sequences_1, alignd_sequences_2)])
  23.     combined_sequences_2 = Concatenate()(
  24.         [encode_sequences_2, alignd_sequences_1, submult(encode_sequences_2, alignd_sequences_1)])
  25.  
  26.     compose = Bidirectional(LSTM(num_lstm, return_sequences=True))
  27.     compare_sequences_1 = compose(combined_sequences_1)
  28.     compare_sequences_2 = compose(combined_sequences_2)
  29.  
  30.     # aggregate
  31.     rep_sequences_1 = apply_multiple(compare_sequences_1, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  32.     rep_sequences_2 = apply_multiple(compare_sequences_2, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  33.  
  34.     # classifier
  35.     merged = Concatenate()([rep_sequences_1, rep_sequences_2])
  36.     dense = BatchNormalization()(merged)
  37.     dense = Dense(num_dense, activation='elu')(dense)
  38.     dense = BatchNormalization()(dense)
  39.     dense = Dropout(rate_drop_dense)(dense)
  40.     dense = Dense(num_dense, activation='elu')(dense)
  41.     dense = BatchNormalization()(dense)
  42.     dense = Dropout(rate_drop_dense)(dense)
  43.     out_ = Dense(1, activation='sigmoid')(dense)
  44.  
  45.     model = Model(inputs=[sequence_1_input, sequence_2_input], outputs=out_)
  46.     model.compile(optimizer=Adam(lr=1e-3), loss='binary_crossentropy', metrics=['binary_crossentropy', 'accuracy'])
  47.     return model
  1. 4. DSSM实现文本相似度
  2. def get_model(embedding_matrix, nb_words, EMBEDDING_DIM, MAX_SEQUENCE_LENGTH, num_lstm, rate_drop_dense):
  3.  
  4.  
  5.     att1_layer = Attention.Attention(MAX_SEQUENCE_LENGTH)
  6.  
  7.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')   # 编码后的问题1的词特征
  8.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')   # 编码后的问题2的词特征
  9.  
  10.     # embedding
  11.     embedding_layer = Embedding(nb_words,
  12.                                 EMBEDDING_DIM,
  13.                                 weights=[embedding_matrix],
  14.                                 input_length=MAX_SEQUENCE_LENGTH,
  15.                                 trainable=False)
  16.     embedded_sequences_1 = embedding_layer(sequence_1_input)
  17.     embedded_sequences_2 = embedding_layer(sequence_2_input)
  18.  
  19.     # encode
  20.     lstm1_layer = Bidirectional(LSTM(num_lstm))
  21.     encode_sequences_1 = lstm1_layer(embedded_sequences_1)
  22.     encode_sequences_2 = lstm1_layer(embedded_sequences_2)
  23.  
  24.     # lstm
  25.     lstm0_layer = LSTM(num_lstm, return_sequences=True)
  26.     lstm2_layer = LSTM(num_lstm)
  27.     v1ls = lstm2_layer(lstm0_layer(embedded_sequences_1))
  28.     v2ls = lstm2_layer(lstm0_layer(embedded_sequences_2))
  29.     v1 = Concatenate(axis=1)([att1_layer(embedded_sequences_1), encode_sequences_1])
  30.     v2 = Concatenate(axis=1)([att1_layer(embedded_sequences_2), encode_sequences_2])
  31.  
  32.     # sequence_1c_input = Input(shape=(MAX_SEQUENCE_LENGTH_CHAR,), dtype='int32')  # 编码后的问题1的字特征
  33.     # sequence_2c_input = Input(shape=(MAX_SEQUENCE_LENGTH_CHAR,), dtype='int32')  # 编码后的问题2的字特征
  34.  
  35.     # embedding_char_layer = Embedding(char_words,
  36.     #                             EMBEDDING_DIM)
  37.  
  38.     # embedded_sequences_1c = embedding_char_layer(sequence_1c_input)
  39.     # embedded_sequences_2c = embedding_char_layer(sequence_2c_input)
  40.  
  41.     # x1c = lstm1_layer(embedded_sequences_1c)
  42.     # x2c = lstm1_layer(embedded_sequences_2c)
  43.     # v1c = Concatenate(axis=1)([att1_layer(embedded_sequences_1c), x1c])
  44.     # v2c = Concatenate(axis=1)([att1_layer(embedded_sequences_2c), x2c])
  45.  
  46.     # compose
  47.     mul = Multiply()([v1, v2])
  48.     sub = Lambda(lambda x: K.abs(x))(Subtract()([v1, v2]))
  49.     maximum = Maximum()([Multiply()([v1, v1]), Multiply()([v2, v2])])
  50.     # mulc = Multiply()([v1c, v2c])
  51.     # subc = Lambda(lambda x: K.abs(x))(Subtract()([v1c, v2c]))
  52.     # maximumc = Maximum()([Multiply()([v1c, v1c]), Multiply()([v2c, v2c])])
  53.     sub2 = Lambda(lambda x: K.abs(x))(Subtract()([v1ls, v2ls]))
  54.     # matchlist = Concatenate(axis=1)([mul, sub, mulc, subc, maximum, maximumc, sub2])
  55.     matchlist = Concatenate(axis=1)([mul, sub, maximum, sub2])
  56.     matchlist = Dropout(rate_drop_dense)(matchlist)
  57.  
  58.     matchlist = Concatenate(axis=1)(
  59.         [Dense(32, activation='relu')(matchlist), Dense(48, activation='sigmoid')(matchlist)])
  60.     res = Dense(1, activation='sigmoid')(matchlist)
  61.  
  62.     # model = Model(inputs=[sequence_1_input, sequence_2_input,
  63.     #                       sequence_1c_input, sequence_2c_input], outputs=res)
  64.     model = Model(inputs=[sequence_1_input, sequence_2_input], outputs=res)
  65.     model.compile(optimizer=Adam(lr=0.001), loss="binary_crossentropy", metrics=['acc'])
  66.     model.summary()
  67.     return model

5. Decomption + Attention实现文本相似度

  1. def get_model(embedding_matrix_file, MAX_SEQUENCE_LENGTH,
  2.               rate_drop_projction, num_projction, hidden_projction,
  3.               rate_drop_compare, num_compare,
  4.               rate_drop_dense, num_dense):
  5.  
  6.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  7.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  8.  
  9.     # embedding
  10.     embedding_layer = create_pretrained_embedding(embedding_matrix_file, mask_zero=False)
  11.     embedded_sequences_1 = embedding_layer(sequence_1_input)
  12.     embedded_sequences_2 = embedding_layer(sequence_2_input)
  13.  
  14.     # projection
  15.     projection_layers = []
  16.     if hidden_projction > 0:
  17.         projection_layers.extend([
  18.             Dense(hidden_projction, activation='elu'),
  19.             Dropout(rate=rate_drop_projction),
  20.         ])
  21.     projection_layers.extend([
  22.         Dense(num_projction, activation=None),
  23.         Dropout(rate=rate_drop_projction),
  24.     ])
  25.     encode_sequences_1 = time_distributed(embedded_sequences_1, projection_layers)
  26.     encode_sequences_2 = time_distributed(embedded_sequences_2, projection_layers)
  27.  
  28.     # attention
  29.     alignd_sequences_1, alignd_sequences_2 = soft_attention_alignment(encode_sequences_1, encode_sequences_2)
  30.  
  31.     # compare
  32.     combined_sequences_1 = Concatenate()(
  33.         [encode_sequences_1, alignd_sequences_2, submult(encode_sequences_1, alignd_sequences_2)])
  34.     combined_sequences_2 = Concatenate()(
  35.         [encode_sequences_2, alignd_sequences_1, submult(encode_sequences_2, alignd_sequences_1)])
  36.     compare_layers = [
  37.         Dense(num_compare, activation='elu'),
  38.         Dropout(rate_drop_compare),
  39.         Dense(num_compare, activation='elu'),
  40.         Dropout(rate_drop_compare),
  41.     ]
  42.     compare_sequences_1 = time_distributed(combined_sequences_1, compare_layers)
  43.     compare_sequences_2 = time_distributed(combined_sequences_2, compare_layers)
  44.  
  45.     # aggregate
  46.     rep_sequences_1 = apply_multiple(compare_sequences_1, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  47.     rep_sequences_2 = apply_multiple(compare_sequences_2, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  48.  
  49.     # classifier
  50.     merged = Concatenate()([rep_sequences_1, rep_sequences_2])
  51.     dense = BatchNormalization()(merged)
  52.     dense = Dense(num_dense, activation='elu')(dense)
  53.     dense = Dropout(rate_drop_dense)(dense)
  54.     dense = BatchNormalization()(dense)
  55.     dense = Dense(num_dense, activation='elu')(dense)
  56.     dense = Dropout(rate_drop_dense)(dense)
  57.     out_ = Dense(1, activation='sigmoid')(dense)
  58.  
  59.     model = Model(inputs=[sequence_1_input, sequence_2_input], outputs=out_)
  60.     model.compile(optimizer=Adam(lr=1e-3), loss='binary_crossentropy', metrics=['binary_crossentropy', 'accuracy'])
  61.     return model
  1. 6. 使用多头自注意力机制的简单网络实现文本相似度
  2. def get_model(embedding_matrix_file, MAX_SEQUENCE_LENGTH, rate_drop_dense):
  3.     sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  4.     sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
  5.  
  6.     # embedding
  7.     embedding_layer = create_pretrained_embedding(embedding_matrix_file, mask_zero=False)
  8.     embedded_sequences_1 = embedding_layer(sequence_1_input)
  9.     embedded_sequences_2 = embedding_layer(sequence_2_input)
  10.  
  11.     # position embedding
  12.     # embedded_sequences_1 = pos_embed.Position_Embedding()(embedded_sequences_1)
  13.     # embedded_sequences_2 = pos_embed.Position_Embedding()(embedded_sequences_2)
  14.  
  15.  
  16.     # attention
  17.     O_seq_1 = Attention.Attention(8, 16)([embedded_sequences_1, embedded_sequences_1, embedded_sequences_1])
  18.     O_seq_2 = Attention.Attention(8, 16)([embedded_sequences_2, embedded_sequences_2, embedded_sequences_2])
  19.  
  20.     # aggregate  ESMI
  21.     # rep_sequences_1 = apply_multiple(compare_sequences_1, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  22.     # rep_sequences_2 = apply_multiple(compare_sequences_2, [GlobalAvgPool1D(), GlobalMaxPool1D()])
  23.  
  24.     rep_sequences_1 = GlobalAveragePooling1D()(O_seq_1)
  25.     rep_sequences_2 = GlobalAveragePooling1D()(O_seq_2)
  26.  
  27.     # classifier
  28.     merged = Concatenate()([rep_sequences_1, rep_sequences_2])
  29.     O_seq = Dropout(rate_drop_dense)(merged)
  30.     outputs = Dense(1, activation='sigmoid')(O_seq)
  31.  
  32.     model = Model(inputs=[sequence_1_input, sequence_2_input], outputs=outputs)
  33.     model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  34.     return model

 

7TF-IDF

TF计算
第三种方案就是直接计算 TF 矩阵中两个向量的相似度了，实际上就是求解两个向量夹角的余弦值，就是点乘积除以二者的模长，公式如下：

cosθ=a·b/|a|*|b|

  1. from sklearn.feature_extraction.text import TfidfVectorizer
  2. import numpy as np
  3. from scipy.linalg import norm
  4.  
  5. def tfidf_similarity(s1, s2):
  6.     def add_space(s):
  7.         return ' '.join(list(s))
  8.  
  9.     # 将字中间加入空格
  10.     s1, s2 = add_space(s1), add_space(s2)
  11.     # 转化为TF矩阵
  12.     cv = TfidfVectorizer(tokenizer=lambda s: s.split())
  13.     corpus = [s1, s2]
  14.     vectors = cv.fit_transform(corpus).toarray()
  15.     # 计算TF系数
  16.     return np.dot(vectors[0], vectors[1]) / (norm(vectors[0]) * norm(vectors[1]))
  17.  
  18. s1 = '你在干嘛呢'
  19. s2 = '你在干什么呢'
  20. print(tfidf_similarity(s1, s2))

这里的 vectors 变量实际上就对应着 TFIDF 值，内容如下：

  1. [[0.         0.         0.4090901  0.4090901  0.57496187 0.4090901 0.4090901 ]
  2.  [0.49844628 0.49844628 0.35464863 0.35464863 0.  0.35464863 0.35464863]]

运行结果如下：

0.5803329846765686

.

相关参考：

https://blog.csdn.net/qq_24140919/article/details/89469318

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/article/detail/42285
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号