Automatic translation from signed to spoken languages is an interdisciplinary research domain, lying on the intersection of computer vision, machine translation and linguistics. Nevertheless, research in this domain is performed mostly by computer scientists in isolation. As the domain is becoming increasingly popular - the majority of scientific papers on the topic of sign language translation have been published in the past three years - we provide an overview of the state of the art as well as some required background in the different related disciplines. We give a high-level introduction to sign language linguistics and machine translation to illustrate the requirements of automatic sign language translation. We present a systematic literature review to illustrate the state of the art in the domain and then, harking back to the requirements, lay out several challenges for future research. We find that significant advances have been made on the shoulders of spoken language machine translation research. However, current approaches are often not linguistically motivated or are not adapted to the different input modality of sign languages. We explore challenges related to the representation of sign language data, the collection of datasets, the need for interdisciplinary research and requirements for moving beyond research, towards applications. Based on our findings, we advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages. Furthermore, the inclusion of deaf and hearing end users of sign language translation applications in use case identification, data collection and evaluation is of the utmost importance in the creation of useful sign language translation models. We recommend iterative, human-in-the-loop, design and development of sign language translation models.
翻译:从手语到口语的自动翻译是一个跨学科的研究领域,涉及计算机视觉、机器翻译和语言的交叉点。然而,这一领域的研究大多由计算机科学家孤立地进行。随着这个领域越来越受欢迎,大多数关于手语翻译主题的科学论文在过去三年中已经发表,我们概述了手语翻译的先进程度以及不同相关学科中某些必要的背景。我们从高级别角度介绍手语语言和机器翻译,以说明自动手语翻译的要求。我们提出系统的文献审查,以说明该领域的先进程度,然后又按照要求进行,提出未来研究面临的若干挑战。我们发现,口语机器翻译的肩膀已经取得重大进展。然而,目前的方法往往没有语言动机,或者没有适应手语语言不同投入模式。我们探讨了手语语言数据的代表性、收集有用的数据集、跨学科研究的需要以及超越研究范围、应用方面的需要。我们根据我们的调查结果,倡导在语言翻译方面进行手势式研究,在语言翻译方面进行最关键的签名,在语言翻译中进行最后的签名。我们主张在语言翻译中进行语言翻译。