Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on eight multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.
翻译:多语种T5(mT5)在大规模单一语言文本的顺序到顺序模式之前就设计了一个大规模单一语言文本的顺序模式,该模式在许多跨语言的任务上已经显示出有希望的成果。在本文件中,我们改进了多语言文本到文本的转换,配有翻译对(mT6),具体地说,我们探索了三种跨语言文本到文本的培训前任务,即机器翻译、双向翻译腐败和翻译腐败。此外,我们提出了文本到文本培训前的部分非自动目标。我们评估了八个多语言基准数据集的方法,包括判决分类、名称实体识别、问题回答和抽象的总结。实验结果表明,拟议的MT6提高了跨语言的可转让性,而不是MT5。