The introduction of pretrained cross-lingual language models brought decisive improvements to multilingual NLP tasks. However, the lack of labelled task data necessitates a variety of methods aiming to close the gap to high-resource languages. Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s). We introduce XeroAlign, a simple method for task-specific alignment of cross-lingual pretrained transformers such as XLM-R. XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages. The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-of-the-art zero-shot results on three multilingual natural language understanding tasks. XLM-RA's text classification accuracy exceeds that of XLM-R trained with labelled data and performs on par with state-of-the-art models on a cross-lingual adversarial paraphrasing task.
翻译:采用经过预先培训的跨语言语言模式对多语言国家语言方案的任务作出了决定性的改进,然而,由于缺少贴标签的任务数据,必须采用各种方法缩小与高资源语言之间的差距。特别是零点方法,常常使用翻译的任务数据作为培训信号,以弥合源和目标语言之间的性能差距。我们引入了XeroAlign,这是一个简单的方法,用于对诸如XLM-R. XeroAlign等跨语言的、经过培训的、经过培训的、经过培训的跨语言的、经过培训的、经过培训的、经过培训的、经过翻译的变压器进行任务调整,以鼓励该模型为不同语言生成类似的句子嵌入。 Xero Along XLM-R,称为XLM-RA,显示在基线模型上有很大改进,以便在三种多种语言的自然语言理解任务中取得最新零点结果。XLM-RA的文本分类精确度超过了经过贴标签数据培训的XLM-R,并在跨语言对抗性参数参数方面与最新模型进行同等的演练。