Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss, thereby improving cross-lingual transferability. Experimental results on mBERT and XLM-R demonstrate that our method significantly outperforms previous works on the zero-shot cross-lingual text classification task and can obtain a better multilingual alignment.
翻译:现有的零点跨语言传输方法依赖于平行的社团或双语词典,对于低资源语言来说,这些词典费用昂贵且不切实际。为了脱离这些依赖关系,研究人员探索了就只使用英语的资源培训多语种模式并将其转移到低资源语言。然而,其效果因不同语言嵌入组之间的差距而受到限制。为了解决这一问题,我们提议将嵌入-普什、注意-普尔和硬性目标将英语嵌入转移到虚拟的多语种嵌入中,而不造成语义损失,从而改进跨语言的可转移性。关于 mBERT和XLM-R的实验结果表明,我们的方法大大优于以前关于零点跨语言文本分类的工作,并能够实现更好的多语种一致性。