We present a new approach for synthesizing novel views of people in new poses. Our novel differentiable renderer enables the synthesis of highly realistic images from any viewpoint. Rather than operating over mesh-based structures, our renderer makes use of diffuse Gaussian primitives that directly represent the underlying skeletal structure of a human. Rendering these primitives gives results in a high-dimensional latent image, which is then transformed into an RGB image by a decoder network. The formulation gives rise to a fully differentiable framework that can be trained end-to-end. We demonstrate the effectiveness of our approach to image reconstruction on both the Human3.6M and Panoptic Studio datasets. We show how our approach can be used for motion transfer between individuals; novel view synthesis of individuals captured from just a single camera; to synthesize individuals from any virtual viewpoint; and to re-render people in novel poses. Code and video results are available at https://github.com/GuillaumeRochette/HumanViewSynthesis.
翻译:使用可微渲染的方法合成人体的新视角。我们提出了一种新的方法,可以合成新的姿态下的人体图像。我们的可微分渲染器可以从任何角度合成高度逼真的图像。与基于网格的结构不同,我们的渲染器使用直接表示人体骨架结构的漫反射高斯基元。渲染这些基元可以得到一个高维潜在图像,然后由解码器网络将其转换为RGB图像。这种表达方式产生了一个完全可微分的框架,可以进行端到端的训练。我们在Human3.6M和Panoptic Studio数据集上展示了我们方法的图像重建效果。我们展示了如何使用我们的方法在个体间进行运动转移,如何从单个摄像机捕获的人体图像中合成新的视角,如何从任意虚拟视角合成人体图像,以及如何在新姿势下重新渲染人体图像。代码和视频结果可在https://github.com/GuillaumeRochette/HumanViewSynthesis找到。