This paper presents the first significant object detection framework, NeRF-RPN, which directly operates on NeRF. Given a pre-trained NeRF model, NeRF-RPN aims to detect all bounding boxes of objects in a scene. By exploiting a novel voxel representation that incorporates multi-scale 3D neural volumetric features, we demonstrate it is possible to regress the 3D bounding boxes of objects in NeRF directly without rendering the NeRF at any viewpoint. NeRF-RPN is a general framework and can be applied to detect objects without class labels. We experimented the NeRF-RPN with various backbone architectures, RPN head designs and loss functions. All of them can be trained in an end-to-end manner to estimate high quality 3D bounding boxes. To facilitate future research in object detection for NeRF, we built a new benchmark dataset which consists of both synthetic and real-world data with careful labeling and clean up. Please watch the \href{https://youtu.be/M8_4Ih1CJjE}{video} for visualizing the 3D region proposals by our NeRF-RPN. Code and dataset will be made available.
翻译:本文介绍了第一个重要物体探测框架NERF-RPN。 NERF-RPN是第一个重要物体探测框架,在NERF上直接运行。根据预先培训的NERF模型,NERF-RPN的目标是在现场探测所有捆绑的物体。通过利用包含多尺寸 3D 神经体积特征的新型反毒说明,我们证明可以直接将NERF中的3D捆绑定的物体复制过来,而不会从任何角度将NERF转换为NERF。NRF-RPN是一个一般框架,可以用于探测没有等级标签的物体。我们用各种主干结构、RPN头设计和损失功能试验NERF-RPN。所有物体都可以接受端到端培训,以估计高质量 3D捆绑定框。为了便利未来对NRFF的物体探测,我们建立了一套新的基准数据集,由合成和真实世界数据组成,仔细标签和清理。请看\href{https://yotu.be/M8_4IH1CJN ⁇ viview}我们国家数据库提供的3D区域的建议。