Document similarity is an important part of Natural Language Processing and is most commonly used for plagiarism-detection and text summarization. Thus, finding the overall most effective document similarity algorithm could have a major positive impact on the field of Natural Language Processing. This report sets out to examine the numerous document similarity algorithms, and determine which ones are the most useful. It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms: statistical algorithms, neural networks, and corpus/knowledge-based algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations that test every possible area that each algorithm could be used in.
翻译:文档相似度是自然语言处理的重要组成部分,最常用于检测抄袭和文本摘要。因此,找到总体上最有效的文档相似度算法可能对自然语言处理领域产生重大积极影响。本文旨在研究众多的文档相似度算法,并确定其中哪些是最有用的。我们将文档相似度算法分类为3种类型:统计算法,神经网络和基于语料库/知识的算法,进而比较各类别中最有效的算法。我们使用一系列基准数据集和评估来测试每种算法可能用到的所有领域,并对每种算法进行比较。