Large Transformer-based language models such as BERT have led to broad performance improvements on many NLP tasks. Domain-specific variants of these models have demonstrated excellent performance on a variety of specialised tasks. In legal NLP, BERT-based models have led to new state-of-the-art results on multiple tasks. The exploration of these models has demonstrated the importance of capturing the specificity of the legal language and its vocabulary. However, such approaches suffer from high computational costs, leading to a higher ecological impact and lower accessibility. Our findings, focusing on English language legal text, show that lightweight LSTM-based Language Models are able to capture enough information from a small legal text pretraining corpus and achieve excellent performance on short legal text classification tasks. This is achieved with a significantly reduced computational overhead compared to BERT-based models. However, our method also shows degraded performance on a more complex task, multi-label classification of longer documents, highlighting the limitations of this lightweight approach.
翻译:大型变异语言模型,如BERT等大型变异语言模型,已导致许多非法律语言项目任务取得广泛的绩效改进,这些模型的域别变异模式在各种专门任务方面表现优异,在法律的NLP中,基于BERT的模型在多重任务方面产生了新的最新成果,这些模型的探索表明捕捉法律语言及其词汇的特殊性的重要性,然而,这些模型的计算成本很高,导致生态影响更大,可获取性更低。我们以英文法律文本为重点的研究结果显示,轻量LSTM语言模型能够从一个小型法律文本预科中获取足够的信息,并在短期法律文本分类任务上取得优异性业绩。这是随着与基于BERT的模型相比计算间接费用大幅减少而实现的。然而,我们的方法还表明,在一项更为复杂的任务上,对较长的文件进行多标签分类的绩效下降,突出了这种轻度方法的局限性。