Automatic translation systems are known to struggle with rare words. Among these, named entities (NEs) and domain-specific terms are crucial, since errors in their translation can lead to severe meaning distortions. Despite their importance, previous speech translation (ST) studies have neglected them, also due to the dearth of publicly available resources tailored to their specific evaluation. To fill this gap, we i) present the first systematic analysis of the behavior of state-of-the-art ST systems in translating NEs and terminology, and ii) release NEuRoparl-ST, a novel benchmark built from European Parliament speeches annotated with NEs and terminology. Our experiments on the three language directions covered by our benchmark (en->es/fr/it) show that ST systems correctly translate 75-80% of terms and 65-70% of NEs, with very low performance (37-40%) on person names.
翻译:已知自动翻译系统会用稀有的词来挣扎。 其中,命名实体(NES)和具体域名术语至关重要,因为翻译中的错误可能导致严重的含义扭曲。 尽管它们很重要,但先前的语音翻译(ST)研究忽视了它们,这也是因为缺乏适合其具体评价的公开资源。为了填补这一空白,我们i)首次对最新科技系统在翻译NES和术语方面的行为进行了系统化分析,二)发布NEuRoparl-ST,这是欧洲议会演讲中设定的带有NES和术语的新基准。我们对基准(en->es/fr/it)所涵盖的三种语言方向的实验表明,ST系统正确翻译了75-80%的术语和65-70%的NES,在个人姓名上表现非常低(37-40% )。