Word embeddings represent words in a numeric space so that semantic relations between words are represented as distances and directions in the vector space. Cross-lingual word embeddings transform vector spaces of different languages so that similar words are aligned. This is done by constructing a mapping between vector spaces of two languages or learning a joint vector space for multiple languages. Cross-lingual embeddings can be used to transfer machine learning models between languages, thereby compensating for insufficient data in less-resourced languages. We use cross-lingual word embeddings to transfer machine learning prediction models for Twitter sentiment between 13 languages. We focus on two transfer mechanisms that recently show superior transfer performance. The first mechanism uses the trained models whose input is the joint numerical space for many languages as implemented in the LASER library. The second mechanism uses large pretrained multilingual BERT language models. Our experiments show that the transfer of models between similar languages is sensible, even with no target language data. The performance of cross-lingual models obtained with the multilingual BERT and LASER library is comparable, and the differences are language-dependent. The transfer with CroSloEngual BERT, pretrained on only three languages, is superior on these and some closely related languages.
翻译:嵌入的字词代表了数字空间中的文字,这样字词之间的语义关系就代表着矢量空间中的距离和方向。跨语言嵌入使不同语言的矢量空间发生转变,从而对相似的字句进行对齐。这是通过在两种语言的矢量空间之间绘制地图,或者为多种语言学习一个共同的矢量空间来完成的。跨语言嵌入可以用来传输不同语言之间的机器学习模型,从而弥补以较少资源的语言提供的数据不足。我们使用跨语言嵌入来传输13种语言之间推特情绪的机器学习预测模型。我们侧重于最近显示高调性能的两个传输机制。第一个机制使用经过培训的模型,其投入是LASER图书馆中实施的许多语言的联合数字空间。第二个机制使用大型事先经过培训的多语言BERT语言模型。我们的实验表明,类似语言之间的模型转换是明智的,即使没有目标语言数据。与多语言BERT和LASER图书馆获得的跨语言模型的性能比较,而且差异也取决于语言。与CroSlongual BERTERT仅受三种语言训练的某种语言的转移是高级的。