Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion. Recent NeRF based methods achieve impressive fidelity of 3D reconstruction, but bake the lighting and shadows into the radiance field, while mesh-based methods that facilitate intrinsic decomposition through differentiable rendering have not yet scaled to the complexity and scale of outdoor scenes. We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows. By faithfully disentangling complex geometry and materials from lighting effects, our method enables photorealistic relighting with specular and shadow effects on several outdoor datasets. Moreover, it supports physics-based scene manipulations such as virtual object insertion with ray-traced shadow casting.
翻译:重建和内在分解被捕获图像的场景将会使许多应用成为可能,如再照明和虚拟物体插入。最近基于NeRF的方法实现了3D重建的令人印象深刻的保真度,但将照明和阴影融入了辐射场中,而通过可微分渲染促进内在分解的基于网格的方法尚未扩展到复杂室外场景的规模。我们提出了一种新颖的反渲染框架,用于大型城市场景,能够从一组位置RGB图像(可以选择深度)中联合重建场景几何、空间可变材料和HDR照明。具体来说,我们使用神经场来描述主光线,并使用显式网格(从基础神经场重建)来建模产生更高阶照明效果的次要光线,例如投射阴影。通过忠实地将复杂几何和材料从照明效果中分离出来,我们的方法可以在几个室外数据集上实现带有镜面和阴影效果的逼真再照明。此外,它支持基于物理的场景操作,例如射线跟踪阴影投射的虚拟物体插入。