In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression and headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation and head translation), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the state-of-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse.
翻译:在本文中,我们提议了一个互动面视频编码的新框架(IFVC),让人类能够与内在视觉代表互动,而不是信号。拟议解决方案具有若干不同优势,包括超方形代表、低延迟互动以及生动表达和头部动画。特别是,我们提议基于内部维度增加(IDI)的表述,极大地提高外观的忠诚度和灵活性,同时保持合理的代表性成本。通过利用强大的统计规律性,视觉信号可以有效地投向三维空间(如口部运动、眨眼、头部旋转和头部翻译)的可控制语义,这三维空间是压缩和传输的。可编辑的位流(这自然支持语义层面的互动性),可以通过深层感化模型的强烈推理能力将面框综合起来。实验结果显示了我们提议的FVVC计划的业绩优势和应用前景。 特别是,拟议的计划不仅超越了最先进的视频编码标准 Versatile 视频编码(VC) 、头部旋转、头部旋转旋转和头部翻译(HIVC) 翻译和头部翻译等翻译的翻译。 编辑可自然地支持语层次互动结构设计框架,还使得最新的影化未来的影质化计划能够将新的影质化框架引入。