While the recent advances in research on video reenactment have yielded promising results, the approaches fall short in capturing the fine, detailed, and expressive facial features (e.g., lip-pressing, mouth puckering, mouth gaping, and wrinkles) which are crucial in generating realistic animated face videos. To this end, we propose an end-to-end expressive face video encoding approach that facilitates data-efficient high-quality video re-synthesis by optimizing low-dimensional edits of a single Identity-latent. The approach builds on StyleGAN2 image inversion and multi-stage non-linear latent-space editing to generate videos that are nearly comparable to input videos. While existing StyleGAN latent-based editing techniques focus on simply generating plausible edits of static images, we automate the latent-space editing to capture the fine expressive facial deformations in a sequence of frames using an encoding that resides in the Style-latent-space (StyleSpace) of StyleGAN2. The encoding thus obtained could be super-imposed on a single Identity-latent to facilitate re-enactment of face videos at $1024^2$. The proposed framework economically captures face identity, head-pose, and complex expressive facial motions at fine levels, and thereby bypasses training, person modeling, dependence on landmarks/ keypoints, and low-resolution synthesis which tend to hamper most re-enactment approaches. The approach is designed with maximum data efficiency, where a single $W+$ latent and 35 parameters per frame enable high-fidelity video rendering. This pipeline can also be used for puppeteering (i.e., motion transfer).
翻译:虽然最近对视频再演化的研究取得了令人乐观的进展,但方法在捕捉细细、详细和表情面部特征(如唇压、口腔拉紧、口隔开和皱纹)方面进展不尽如人意,这些特征对于产生现实的动画脸视频至关重要。为此,我们建议采用端到端的表情图像编码方法,通过优化单一身份latient(SteleGAN2)的低维编辑方式,促进数据高效的高质量视频再合成。SteleGAN2的低维度编辑方法基于SteleGAN2图像的倒置和多级的非线性潜伏空间编辑方法,以生成几乎与输入视频相近的视频。虽然现有的StelegGAN的隐性编辑技术侧重于仅仅生成真实的静态图像修改,但我们将潜空的图像编辑方法自动化,以便利用Steleg-lat-lat-space(Stylespace)中包含的编码,由此获得的母体编码,从而可以在一个单一身份-相对的图像转换和最深层直径直径直径的图像框架上转换的图像转换工具上,从而在1024上进行快速的图像的图像分析。