Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.
翻译:自然图像是 2D 图像平面上的 3D 对象的投影。 虽然像 GANs 这样的最先进的 2D 基因化模型在模拟自然图像元件方面表现出前所未有的质量, 但尚不清楚这些模型是否暗含地捕捉了3D 对象结构。 如果是这样, 我们如何利用这些知识来恢复图像中3D 对象的3D 形状? 为了回答这些问题, 我们在此工作中首次尝试直接从一个仅用 RGB 图像培训的 3D GAN 上直接挖掘3D 的3D 几何线索。 我们通过调查发现, 这种经过训练的 GAN 模型确实包含丰富的 3D 知识, 从而可以用未受监督的方式从一个 2D 对象图像中恢复 3D 形状 。 我们框架的核心是一个互动战略, 探索和利用 GAN 图像的多种观点和照明变异。 这个框架不需要 2D 关键点或 3D 说明, 或对对象形状( 例如 形状是对称) 。 然而, 我们的3D 形状成功地恢复了 3D 形状, 以高精度的模型, 以高精度的模型来进行我们 的 。