Reconstructing 3D layouts from multiple $360^{\circ}$ panoramas has received increasing attention recently as estimating a complete layout of a large-scale and complex room from a single panorama is very difficult. The state-of-the-art method, called PSMNet, introduces the first learning-based framework that jointly estimates the room layout and registration given a pair of panoramas. However, PSMNet relies on an approximate (i.e., "noisy") registration as input. Obtaining this input requires a solution for wide baseline registration which is a challenging problem. In this work, we present a complete multi-view panoramic layout estimation framework that jointly learns panorama registration and layout estimation given a pair of panoramas without relying on a pose prior. The major improvement over PSMNet comes from a novel Geometry-aware Panorama Registration Network or GPR-Net that effectively tackles the wide baseline registration problem by exploiting the layout geometry and computing fine-grained correspondences on the layout boundaries, instead of the global pixel-space. Our architecture consists of two parts. First, given two panoramas, we adopt a vision transformer to learn a set of 1D horizon features sampled on the panorama. These 1D horizon features encode the depths of individual layout boundary samples and the correspondence and covisibility maps between layout boundaries. We then exploit a non-linear registration module to convert these 1D horizon features into a set of corresponding 2D boundary points on the layout. Finally, we estimate the final relative camera pose via RANSAC and obtain the complete layout simply by taking the union of registered layouts. Experimental results indicate that our method achieves state-of-the-art performance in both panorama registration and layout estimation on a large-scale indoor panorama dataset ZInD.
翻译:重新构建由多个 360 ⁇ circ}$ panorama 提供的 3D 版面布局最近越来越受到越来越多的关注,因为从一个全景中估算一个大型和复杂房间的完整布局非常困难。 最先进的方法叫做 PSMNet, 引入了第一个基于学习的框架, 共同估算房间布局和注册, 给一对全色。 然而, PSMNet 依靠一个近似( e., “ noisy ” ) 的注册作为输入。 获得这一输入需要找到一个解决方案, 用于广泛的直线登记。 这是一个具有挑战性的问题。 在这项工作中, 我们展示了一个完整的多视图全景全色和复杂房间的版面布局估计框架框架, 共同学习全色的登记和布局估计, 而不依赖于一个全色的全色版面的全色版面布局。 我们的架构由两部的直观地平面的内地平面布局运行结果组成了两个部分 。 我们的图将这两组的直地平面的直径直径布局的平面的平面的布局, 将两个直径布局的平面的底图 标的底图 向的底图显示为两个部的平面图 。