A key challenge for novel view synthesis of monocular portrait images is 3D consistency under continuous pose variations. Most existing methods rely on 2D generative models which often leads to obvious 3D inconsistency artifacts. We present a 3D-consistent novel view synthesis approach for monocular portrait images based on a recent proposed 3D-aware GAN, namely Generative Radiance Manifolds (GRAM), which has shown strong 3D consistency at multiview image generation of virtual subjects via the radiance manifolds representation. However, simply learning an encoder to map a real image into the latent space of GRAM can only reconstruct coarse radiance manifolds without faithful fine details, while improving the reconstruction fidelity via instance-specific optimization is time-consuming. We introduce a novel detail manifolds reconstructor to learn 3D-consistent fine details on the radiance manifolds from monocular images, and combine them with the coarse radiance manifolds for high-fidelity reconstruction. The 3D priors derived from the coarse radiance manifolds are used to regulate the learned details to ensure reasonable synthesized results at novel views. Trained on in-the-wild 2D images, our method achieves high-fidelity and 3D-consistent portrait synthesis largely outperforming the prior art.
翻译:单切肖像像的新视角合成的关键挑战在于连续的3D一致性。大多数现有方法都依赖于2D基因模型,这些模型往往导致明显的3D不一致的文物。我们根据最近提出的3D-aware GAN, 即GRAM(GRAM),对单切肖像像进行三维一致的新视角合成方法, 即3D-aware GAN(GRAM), 它显示在通过光亮柱代表的虚拟对象的多维图像生成中, 3D一致性很强。然而, 仅仅学习一个编码器将真实图像映射到 GRAM 的潜藏空间, 只能在没有忠实的精细细节的情况下重建粗浅的光柱形元, 而通过具体实例的优化来改善重建对单切肖像图像的忠实性。 我们引入了一个新颖的三维相容元重建元集, 并把它们与高纤维质重建的粗利的弧度弧度柱结合起来。 由粗利晶的光柱子生成的三维维, 用于调节所学的细节, 以确保新颖的合成结果的合成结果的合成, 3D 之前的合成法, 基本化的合成法 。