3D-aware GANs aim to synthesize realistic 3D scenes such that they can be rendered in arbitrary perspectives to produce images. Although previous methods produce realistic images, they suffer from unstable training or degenerate solutions where the 3D geometry is unnatural. We hypothesize that the 3D geometry is underdetermined due to the insufficient constraint, i.e., being classified as real image to the discriminator is not enough. To solve this problem, we propose to approximate the background as a spherical surface and represent a scene as a union of the foreground placed in the sphere and the thin spherical background. It reduces the degree of freedom in the background field. Accordingly, we modify the volume rendering equation and incorporate dedicated constraints to design a novel 3D-aware GAN framework named BallGAN. BallGAN has multiple advantages as follows. 1) It produces more reasonable 3D geometry; the images of a scene across different viewpoints have better photometric consistency and fidelity than the state-of-the-art methods. 2) The training becomes much more stable. 3) The foreground can be separately rendered on top of different arbitrary backgrounds.
翻译:3D-aware GANs 旨在将现实的 3D 场景合成为现实的 3D 场景, 以便以任意的视角制作图像。 虽然以前的方法产生了现实的图像, 但是在 3D 几何不自然的地方, 它们会受到不稳定的训练或退化的解决方案。 我们假设3D 几何由于限制不足, 也就是说, 被歧视者归类为真实的图像是不够的。 为了解决这个问题, 我们提议将背景相近为球形表面, 并代表一个场景作为位于球体和稀薄球形背景的地表层的组合。 它会降低背景领域的自由度。 因此, 我们修改体积的方程式, 并纳入专门的限制来设计名为 BallGAN 的3D-awa GAN 的新的3D GAN 框架。 BallGAN 有多种优点 。 1) 它产生更合理的 3D 几何形状; 不同角度的场景图像比状态方法更具有光度的一致性和准确性。 2) 培训变得更加稳定。 (3) 地面可以在不同任意背景的顶部上分别设置。