Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i)~the rigid transformation (scale, rotation, and translation) and (ii)~the non-rigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we use the generalized Student t-distribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. The method is thoroughly evaluated and compared with several state of the art methods, either based on traditional geometric models or on deep learning. Moreover, we show that the method, when incorporated into deep learning pipelines, namely lip reading and speech enhancement, improves word recognition and speech intelligibilty scores by a considerable margin. Supplemental material is accessible at https://team.inria.fr/robotlearn/research/facefrontalization/
翻译:正面面面部的变形包括从任意查看的图像中合成前视面部的图像。 本文的主要贡献是使用前视方法, 以保存非硬面部变形, 以提高视觉辅助语音通信的性能。 估计( i) 至死硬的变形( 缩略、 旋转和翻译) 和 (ii) 至 至 至 至, 非硬面部变形, 由任意观看的面部和面部模型进行。 这种方法有两个重要优点: 它可以处理数据中的非加西文错误, 并包含动态面部变形模型。 为此, 我们使用普世学生T分配与线性动态系统相结合的方法, 以便既考虑到硬头部运动, 也考虑到由语音制作造成的时间变化。 我们提议使用零度的正常交叉关系( ZNCC) 评分来评价保持面部表达方式的能力。 该方法经过彻底评估, 并与几种艺术方法的状态进行比较, 要么基于传统的几何计量模型, 要么是在深层语音/ 上学习 。