Pretrained transformer models have achieved state-of-the-art results in many tasks and benchmarks recently. Many state-of-the-art Language Models (LMs), however, do not scale well above the threshold of 512 input tokens. In specialized domains though (such as legal, scientific or biomedical), models often need to process very long text (sometimes well above 10000 tokens). Even though many efficient transformers have been proposed (such as Longformer, BigBird or FNet), so far, only very few such efficient models are available for specialized domains. Additionally, since the pretraining process is extremely costly in general - but even more so as the sequence length increases - it is often only in reach of large research labs. One way of making pretraining cheaper is the Replaced Token Detection (RTD) task, by providing more signal during training, since the loss can be computed over all tokens. In this work, we train Longformer models with the efficient RTD task on legal data to showcase that pretraining efficient LMs is possible using much less compute. We evaluate the trained models on challenging summarization tasks requiring the model to summarize long texts to show to what extent the models can achieve good performance on downstream tasks. We find that both the small and base models outperform their baselines on the in-domain BillSum and out-of-domain PubMed tasks in their respective parameter range. We publish our code and models for research purposes.
翻译:最近许多任务和基准都取得了最先进的变压器模型。然而,许多最先进的语文模型(LMS)并没有大大超过512个输入标牌的门槛。在专门领域(法律、科学或生物医学等),模型往往需要处理非常长的文本(有时大大高于10 000个标牌 ) 。尽管提出了许多高效的变压器(如Longfore、BigBird或FNet ),但迄今为止,在专门领域只有很少这样的高效模型。此外,由于培训前过程一般费用极高,但随着序列长度的延长,培训前的模型往往更昂贵,因此,它往往仅仅达到512个输入标牌的门槛。在专业领域(例如法律、科学或生物医学),使培训前更便宜的一个方法是在培训期间提供更多的信号,因为损失可以按所有标牌计算。在法律数据方面,我们用高效的变压模型来显示培训前的LMMS是可能的。我们评估了具有挑战性的总和精细的模型的模型。我们用模型来总结其底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底底。我们能。我们能。我们用到一定范围。我们用到一定范围。我们用得得得到一个大小。