Novel view synthesis (NVS) is a challenging task requiring systems to generate photorealistic images of scenes from new viewpoints, where both quality and speed are important for applications. Previous image-based rendering (IBR) methods are fast, but have poor quality when input views are sparse. Recent Neural Radiance Fields (NeRF) and generalizable variants give impressive results but are not real-time. In our paper, we propose a generalizable NVS method with sparse inputs, called FWD, which gives high-quality synthesis in real-time. With explicit depth and differentiable rendering, it achieves competitive results to the SOTA methods with 130-1000x speedup and better perceptual quality. If available, we can seamlessly integrate sensor depth during either training or inference to improve image quality while retaining real-time speed. With the growing prevalence of depths sensors, we hope that methods making use of depth will become increasingly useful.
翻译:新观点合成(NVS)是一项具有挑战性的任务,要求各系统从新角度产生图像真实的图像,这些图像的质量和速度对于应用都很重要。以前基于图像的拍摄方法(IBR)很快,但当输入视图稀少时质量差。最近神经辐射场(NERF)和通用变体(NERF)带来了令人印象深刻的结果,但不是实时的。在我们的论文中,我们提出了一个通用的NVS方法,其投入稀少,称为FWD,它能实时提供高质量的合成。如果深度和速度明显不同,它能够以130-1000x的速度和更好的概念质量实现SOTA方法的竞争效果。如果有的话,我们可以在培训或推断提高图像质量的同时保留实时速度时,无缝地将传感器深度整合。随着深度传感器的日益普及,我们希望利用深度的方法将变得日益有用。