We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for high-quality surface generation in minutes. It marries the implicit volumetric rendering of the truncated signed distance field (TSDF) with multi-resolution hash encoding. We further propose a hybrid neural tracker to generate animated meshes, which combines explicit non-rigid tracking with implicit dynamic deformation in a self-supervised framework. The former provides the coarse warping back into the canonical space, while the latter implicit one further predicts the displacements using the 4D hash encoding as in our reconstructor. Then, we discuss the rendering schemes using the obtained animated meshes, ranging from dynamic texturing to lumigraph rendering under various bandwidth settings. To strike an intricate balance between quality and bandwidth, we propose a hierarchical solution by first rendering 6 virtual views covering the performer and then conducting occlusion-aware neural texture blending. We demonstrate the efficacy of our approach in a variety of mesh-based applications and photo-realistic free-view experiences on various platforms, i.e., inserting virtual human performances into real environments through mobile AR or immersively watching talent shows with VR headsets.
翻译:我们最近看到在光真人模型和成像的神经进步方面取得了巨大的进步。 然而, 将它们融入现有的以网状为基础的下游应用管道仍具有挑战性。 在本文中, 我们提出一个全面的神经进化方法, 用于高质量的重建、压缩和从密集的多视图视频中展示人类的表演。 我们的核心直觉是连接传统动画网形工作流程, 使用新型的高效神经技术。 我们首先引入一个神经表面重建器, 用于在几分钟内进行高质量的地表生成。 但是, 将那些以多分辨率编码的方式将它们整合到一个现有的网状的远程网络中去。 我们进一步提出一个混合神经跟踪器, 来产生动动动的内衣, 将明显的非硬性跟踪与隐含的动态变形变形连接起来。 前者提供了向银河空间的粗微扭曲, 而后一种暗的预言意是进一步预测在我们的重建过程中使用基于4D的变形变形方法进行迁移。 然后, 我们用在虚拟的轨变现模型中, 将一个通过一个动态的正态变色的图像结构显示我们之前的变色的图像显示的图像。