Text Similarity involves measuring the degree of similarity between two or more pieces of text. This can be achieved using various techniques, such as cosine similarity or Jaccard similarity, and is often used in tasks like duplicate detection or information retrieval.

Posts

Unsupervised Concept Representation Learning for Length-Varying Text Similarity

/in /by

Measuring document similarity plays an important role in natural language processing tasks. Most existing document similarity approaches suffer from the information gap caused by context and vocabulary mismatches when comparing varying-length texts. In this paper, we propose an unsupervised concept representation learning approach to address the above issues. Specifically, we propose a novel Concept Generation Network (CGNet) to learn concept representations from the perspective of the entire text corpus. Moreover, a concept-based document matching method is proposed to leverage advances in the recognition of local phrase features and corpus-level concept features. Extensive experiments on real-world data sets demonstrate that new method can achieve a considerable improvement in comparing length-varying texts. In particular, our model achieved 6.5% better F1 Score compared to the best of the baseline models for a concept-project benchmark dataset.

Read more