面向聋儿言语康复的多模态人机交互模型及技术研究

项目名称： 面向聋儿言语康复的多模态人机交互模型及技术研究

项目编号： No.61502052

项目类型： 青年科学基金项目

立项/批准年度： 2016

项目学科： 计算机科学学科

项目作者： 赵剑

作者单位： 长春大学

项目金额： 21万元

中文摘要： 高级人机交互的计算理论研究是国家重点基础研究发展计划，多模态人机交互模型是该领域的研究热点和前沿，真实感三维人脸形变技术是其中的关键技术。本项目提出的特殊人机交互模型多模态包括声音、震动、气流、视频、手势等，通过计算机产生逼真的表情和面部动作，配合与之相对应的语音震动气流等反馈信息，可以更加逼真、全面地对人发音过程进行模拟。运用数据驱动的真实感三维人脸型变技术研究多模态人机交互模型及交互技术，用以解决非人工干预下聋儿自主言语康复训练难题。建立真实感三维人脸形变模型，建立可视与非可视相融合的信息同步算法，实现对真实感问题的真实表达；建立面向聋儿的真实感三维人脸型变汉语发音驱动数据库；建立数据驱动的真实感三维人脸形变模型为核心的聋儿同计算机的多模态人机交互模型；建立基于多模态人机交互模型的聋儿言语康复方法及系统。项目研究成果的应用将使中国百万由聋致贫家庭摆脱贫困，社会效益巨大。

中文关键词： 多模态人机交互；康复工程

英文摘要： Research on computational theory of advanced human-computer interaction is a key basic research planning of nation and Multimodal human-computer interaction is a focus and front branch in this area. Reality Morhpable model based 3D Face Synthesis is the key technology of Multimodal human-computer interaction. The Multimodal human-computer conversation model which proposed by the project concludes the modules of sound, motion, air flow, video, gesture and so on. The facial expressions and movement can be produced by computer, with the voice air flow and their feedback; the human pronunciation process can be simulated more realistically and comprehensive. Data-driven Morhpable model based 3D Face Synthesis is used to build Multimodal human-computer interaction model and interaction technology, the method is used to solve deaf children recovery without artificial intervention.The Morhpable model based 3D Face Synthesis is built. The information synchronization algorithm which merges visual and non-visual part is proposed to realize the true expression of reality sense. The Chinese voice driven database which faced to deaf children is built. The human-computer interaction model is built, the model is driven by data, and focus on Morhpable model based 3D Face Synthesis. The deaf children language recovery method and system which based on Multimodal human-computer interaction model is built. The application of the research in the project will make millions of families which are poverty by deaf children get rid of poverty, it will have great social benefit.

英文关键词： Multimodal Human-computer Interaction;The Rehabilitation Engineering