The recent research explosion around Neural Radiance Fields (NeRFs) shows that there is immense potential for implicitly storing scene and lighting information in neural networks, e.g., for novel view generation. However, one major limitation preventing the widespread use of NeRFs is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS when aiming for real-time rendering on current devices. We show that the number of samples required for each view ray can be significantly reduced when local samples are placed around surfaces in the scene. To this end, we propose a depth oracle network, which predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, a dual network design with a depth oracle network as a first step and a locally sampled shading network for ray accumulation. With our design, we reduce the inference costs by up to 48x compared to NeRF. Using an off-the-shelf inference API in combination with simple compute kernels, we are the first to render raymarching-based neural representations at interactive frame rates (15 frames per second at 800x800) on a single GPU. At the same time, since we focus on the important parts of the scene around surfaces, we achieve equal or better quality compared to NeRF.
翻译:最近围绕神经辐射场(NERFs)的研究爆炸表明,在神经网络中隐蔽地储存场景和照明信息(例如,新视觉生成)的潜力巨大,但是,阻止广泛使用NERFs的一个主要限制是,在每一光线上进行过多的网络评价的计算成本过高,每光线上需要几十个PetaFLOPS来实时显示当前设备。我们显示,当将当地样品放在现场表面周围时,每部观测光线所需的样本数量可以大大减少。为此,我们提议建立一个深度或触角网络,通过单一网络评估,预测每个光线的射线的射线采样位置。我们表明,使用围绕对离异和球面深度值的分类网络,对于对地表位置进行编码,而不是直接估计深度,是至关重要的。这些技术的结合导致DoneRF,一个以深度或触角网络为基础的双网络设计,以及一个地方取样质量阴影网络,以进行射线积累。我们的设计,我们把每个光谱上的摄像标点温度降低至48x重要位置,从我们直径图面的直径直径直到直径直径直径直径直至直径。我们。使用直径直径直径直径直地的图像图图,使用直直直直直至直径直直直直至直直直直直方方方方方方方的图像。