Recommender systems assist legal professionals in finding relevant literature for supporting their case. Despite its importance for the profession, legal applications do not reflect the latest advances in recommender systems and representation learning research. Simultaneously, legal recommender systems are typically evaluated in small-scale user study without any public available benchmark datasets. Thus, these studies have limited reproducibility. To address the gap between research and practice, we explore a set of state-of-the-art document representation methods for the task of retrieving semantically related US case law. We evaluate text-based (e.g., fastText, Transformers), citation-based (e.g., DeepWalk, Poincar\'e), and hybrid methods. We compare in total 27 methods using two silver standards with annotations for 2,964 documents. The silver standards are newly created from Open Case Book and Wikisource and can be reused under an open license facilitating reproducibility. Our experiments show that document representations from averaged fastText word vectors (trained on legal corpora) yield the best results, closely followed by Poincar\'e citation embeddings. Combining fastText and Poincar\'e in a hybrid manner further improves the overall result. Besides the overall performance, we analyze the methods depending on document length, citation count, and the coverage of their recommendations. We make our source code, models, and datasets publicly available at https://github.com/malteos/legal-document-similarity/.
翻译:建议系统协助法律专业人员寻找相关文献来支持其案件。尽管法律应用程序对专业很重要,但法律应用程序并不反映推荐人系统和代表性学习研究的最新进展。与此同时,法律建议系统通常在小规模用户研究中评价,而没有公开的基准数据集。因此,这些研究的可复制性有限。为了解决研究和实践之间的差距,我们探索了一套关于检索与其案件有关的教义法任务的最先进的文件代表方法。我们评估了基于文本的(例如,快通、变换器),基于引用的(例如,DeepWalk、Poincar\e)和混合方法。我们用总共27种方法对小规模用户研究进行评价,没有提供任何公开的基准数据集。银本标准是新根据公开案例书和维基源创建的,可以在公开许可下再利用。我们的实验显示,来自平均快通词汇的词汇矢量(训练于法律公司)的文件表述产生了最佳结果,由Poincar\Poincar\“参考版” 和“快速浏览数据/结果分析方法,我们在快速浏览文件/结果中,我们用快速分析。