With ever-expanding datasets of domains, tasks and languages, transfer learning (TL) from pre-trained neural language models has emerged as a powerful technique over the years. Many pieces of research have shown the effectiveness of transfer learning across different domains and tasks. However, there remains uncertainty around when a transfer will lead to positive or negative impacts on performance of the model. To understand the uncertainty, we investigate how TL affects the performance of popular pre-trained models like BERT, RoBERTa and XLNet over three natural language processing (NLP) tasks. We believe this work will inform about specifics on when and what to transfer related to domain, multi-lingual dataset and various NLP tasks.
翻译:随着领域、任务和语言数据集的不断扩大,多年来,从经过培训的神经语言模型中转移学习(TL)已成为一项强有力的技术,许多研究都表明在不同领域和任务中转移学习的有效性,然而,在何时转移将给模型的绩效带来积极或消极影响方面,还存在着不确定性。为了了解不确定性,我们调查TL如何影响诸如BERT、RoBERTA和XLNet等经过培训的流行型模型在三种自然语言处理任务方面的绩效。我们认为,这项工作将告知何时和什么时间转移与域、多语言数据集和各种非语言处理任务有关的具体信息。