We propose LookinGood^{\pi}, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.
翻译:我们提出一个全新的神经再造方法,旨在(1) 提高人类实时性能捕获系统所重建的低质量结果的质量;(2) 提高对看不见的人的神经转换网络的普及能力。 我们的关键想法是利用重建的几何图像作为指导,帮助从少数参考图像中预测具体个人的细节,从而增强重新产生的结果。 我们为此设计了一个双部门网络。 一个粗糙的分支旨在修复一些人工制品(即洞、噪音)并获得一个粗糙的投入版本,而一个详细的分支旨在预测扭曲引用的参考文献中的“正确”细节。 将两个分支的特征有效地结合到细节分支的培训中,从而实现对图像的引导,从而改进扭曲的准确性和细节的忠诚性。 我们证明我们的方法超越了在制作对看不见的人的高度性能图像方面的最先进的方法。