The best way to combine the results of deep learning with standard 3D reconstruction pipelines remains an open problem. While systems that pass the output of traditional multi-view stereo approaches to a network for regularisation or refinement currently seem to get the best results, it may be preferable to treat deep neural networks as separate components whose results can be probabilistically fused into geometry-based systems. Unfortunately, the error models required to do this type of fusion are not well understood, with many different approaches being put forward. Recently, a few systems have achieved good results by having their networks predict probability distributions rather than single values. We propose using this approach to fuse a learned single-view depth prior into a standard 3D reconstruction system. Our system is capable of incrementally producing dense depth maps for a set of keyframes. We train a deep neural network to predict discrete, nonparametric probability distributions for the depth of each pixel from a single image. We then fuse this "probability volume" with another probability volume based on the photometric consistency between subsequent frames and the keyframe image. We argue that combining the probability volumes from these two sources will result in a volume that is better conditioned. To extract depth maps from the volume, we minimise a cost function that includes a regularisation term based on network predicted surface normals and occlusion boundaries. Through a series of experiments, we demonstrate that each of these components improves the overall performance of the system.
翻译:将深度学习的结果与标准的 3D 重建管道相结合的最佳方法仍然是一个尚未解决的问题。虽然通过传统多视立体法对常规化或完善网络进行常规化或完善的系统目前似乎取得了最佳结果,但最好将深神经网络作为独立的组成部分,其结果可以概率地结合到基于几何的系统中。 不幸的是,进行这种类型的聚合所需的错误模型没有很好地理解,许多不同的方法正在提出。最近,少数系统通过让其网络预测概率分布而不是单一值而取得了良好的结果。我们建议使用这种方法将一个学习的单一视界方法将一个学习的单一视界深度连接到标准 3D 重建系统。我们的系统能够为一组关键框架制作密度的深度地图。我们训练一个深度神经网络,以预测离散的、非直径概率分布,从一个图像中可以提出许多不同的方法。我们然后根据随后的框架和关键框架图像之间的光度一致性将这个“概率体积”与另一个概率体积结合起来。我们主张用这种方法将这两个来源的概率整体性能结合到一个标准3D 重建系统,每个深度的概率量将让我们从一个正常的模型中得出一个数值。