The recent research explosion around implicit neural representations, such as NeRF, shows that there is immense potential for implicitly storing high-quality scene and lighting information in neural networks. However, one major limitation preventing the use of NeRF in interactive and real-time rendering applications is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS when aiming for real-time rendering on consumer hardware. In this work, we take a step towards bringing neural representations closer to practical rendering of synthetic content in interactive and real-time applications, such as games and virtual reality. We show that the number of samples required for each view ray can be significantly reduced when local samples are placed around surfaces in the scene. To this end, we propose a depth oracle network, which predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, a dual network design with a depth oracle network as a first step and a locally sampled shading network for ray accumulation. With our design, we reduce the inference costs by up to 48x compared to NeRF. Using an off-the-shelf inference API in combination with simple compute kernels, we are the first to render raymarching-based neural representations at interactive frame rates (15 frames per second at 800x800) on a single GPU. At the same time, since we focus on the important parts of the scene around surfaces, we achieve equal or better quality compared to NeRF to enable interactive high-quality rendering.
翻译:最近在NeRF等隐性神经显示器周围发生的研究爆炸表明,在神经网络中隐含地储存高质量场景和照明信息的可能性巨大。然而,阻止将NeRF用于互动和实时投影应用的一个重大限制是,每个光线上过度网络评价的计算成本令人望而却步,每个光线上需要数十个花生FLOOPS来实时提供消费硬件。在这项工作中,我们迈出了一步,使神经显示更加接近互动和实时应用程序(如游戏和虚拟现实)中合成内容的实际传输。我们显示,当将当地样品放在现场的表面周围时,每个光线所需的样本数量可以大大减少。为此,我们提议了一个深度或触角网络的深度或触角网络,通过单一的网络评价来预测每个光谱的采样位置。我们用一个分类网络的离谱化和球形深度的深度值,而不是直接估计深度。这些技术的组合导致DONERF,一个带有深度或骨质的双网络设计,其深度或骨质标值可以大大降低。我们从网络的直径直径的深度到直径的直径直径直径网络,从一个直径直到直到直径直到直到直方,从一个直径直到直方,从一个直径方方,从一个直到直到直方方,从一个直方,从一个直径直方,从一个直方到直方,从一个直方到直方到直到直到直到直到直方,从我们方到直到直方到直方到直方到直方到直方到直方,我们方到直方,从一个直方,从一个直到直方,从一个直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方,我们方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直方到直