According to the World Federation of the Deaf, more than two hundred sign languages exist. Therefore, it is challenging to understand deaf individuals, even proficient sign language users, resulting in a barrier between the deaf community and the rest of society. To bridge this language barrier, we propose a novel multilingual communication system, namely MUGCAT, to improve the communication efficiency of sign language users. By converting recognized specific hand gestures into expressive pictures, which is universal usage and language independence, our MUGCAT system significantly helps deaf people convey their thoughts. To overcome the limitation of sign language usage, which is mostly impossible to translate into complete sentences for ordinary people, we propose to reconstruct meaningful sentences from the incomplete translation of sign language. We also measure the semantic similarity of generated sentences with fragmented recognized hand gestures to keep the original meaning. Experimental results show that the proposed system can work in a real-time manner and synthesize exquisite stunning illustrations and meaningful sentences from a few hand gestures of sign language. This proves that our MUGCAT has promising potential in assisting deaf communication.
翻译：据世界聋人联合会称,现有200多种手语,因此,理解聋人,甚至熟练手语使用者,对理解聋人、甚至熟练手语使用者,造成聋人与社会其他人之间的障碍具有挑战性。为了弥合这一语言障碍,我们提议建立一个新型的多语种通信系统,即MUGCAT,以提高手语使用者的沟通效率。通过将公认的具体手势转换为表情图片,即普遍使用和语言独立,我们的MUGCAT系统极大地帮助聋人表达自己的想法。为了克服手语的使用限制,而手语的使用大多无法转换成对普通人的完整句子,我们提议用手语翻译不完整来重塑有意义的句子。我们还用支离破碎的手势势测量生成的语义相似性,以保持原意。实验结果表明,拟议的系统可以实时运作,并综合从几手手手手手手手语手语的精美的插画和有意义的句子。这证明,我们的手语翻译系统在协助聋人沟通方面很有潜力。
相关内容
微信扫码咨询专知VIP会员