This registered report introduces the largest, and for the first time, reproducible experimental survey on biomedical sentence similarity with the following aims: (1) to elucidate the state of the art of the problem; (2) to solve some reproducibility problems preventing the evaluation of most of current methods; (3) to evaluate several unexplored sentence similarity methods; (4) to evaluate an unexplored benchmark, called Corpus-Transcriptional-Regulation; (5) to carry out a study on the impact of the pre-processing stages and Named Entity Recognition (NER) tools on the performance of the sentence similarity methods; and finally, (6) to bridge the lack of reproducibility resources for methods and experiments in this line of research. Our experimental survey is based on a single software platform that is provided with a detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments. In addition, we introduce a new aggregated string-based sentence similarity method, called LiBlock, together with eight variants of current ontology-based methods and a new pre-trained word embedding model trained on the full-text articles in the PMC-BioC corpus. Our experiments show that our novel string-based measure sets the new state of the art on the sentence similarity task in the biomedical domain and significantly outperforms all the methods evaluated herein, except one ontology-based method. Likewise, our experiments confirm that the pre-processing stages, and the choice of the NER tool, have a significant impact on the performance of the sentence similarity methods. We also detail some drawbacks and limitations of current methods, and warn on the need of refining the current benchmarks. Finally, a noticeable finding is that our new string-based method significantly outperforms all state-of-the-art Machine Learning models evaluated herein.
翻译:这份已登记的报告首次介绍了最大的生物医学判决相似性实验性调查,其目标如下:(1) 阐明问题的最新状态;(2) 解决某些阻碍评估目前方法中大多数方法的再生问题;(3) 评估几项尚未探讨的判决相似性方法;(4) 评价一个未探讨的基准,称为Corpus-Transital- Control;(5) 研究预处理阶段和命名实体识别工具对判决相似性绩效的影响;以及最后,(6) 弥补这方面研究中缺乏方法和实验的再生资源的问题;(2) 解决一些妨碍评价目前方法中大多数方法的再生问题;(3) 评估几个未经探讨的类似判决方法;(4) 评估一个未经探讨的基准,称为Corpus-Trancripal-Proformation;(5) 研究前处理阶段和命名实体识别工具(NER)对判决相似性的影响;以及最后,(6) 弥补这方面研究中的方法和实验中经培训的完整版本的再生资源资源的缺乏。