Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.
翻译:旨在提取源与目标句之间等同词汇翻译的词校对,是自然语言处理的基本工具。最近在这一领域的研究通过预先培训的多语文模式的内嵌背景化嵌入,取得了显著的改进。然而,我们发现,现有办法在输入句对配之间几乎没有互动,这严重地降低了单语语语语中模糊词的词校对质量。为了纠正这一问题,我们建议跨对配对模式在输入句之间进行深层互动,其中源和目标句与浅层共享的自我注意模块分别编码,而跨语言互动则由上层的交叉注意模块明确构建。此外,为了有效地培训我们的模型,我们提议了一个两阶段培训框架,在第一阶段对模式进行培训时,以简单的翻译语言建模(TLM)目标为基础,然后在第二阶段与自上调校准的校正目标进行调整。实验显示,拟议的交叉组合在五组语言的四组中实现了状态。