Domain adaptation is critical for success when confronting with the lack of annotations in a new domain. As the huge time consumption of labeling process on 3D point cloud, domain adaptation for 3D semantic segmentation is of great expectation. With the rise of multi-modal datasets, large amount of 2D images are accessible besides 3D point clouds. In light of this, we propose to further leverage 2D data for 3D domain adaptation by intra and inter domain cross modal learning. As for intra-domain cross modal learning, most existing works sample the dense 2D pixel-wise features into the same size with sparse 3D point-wise features, resulting in the abandon of numerous useful 2D features. To address this problem, we propose Dynamic sparse-to-dense Cross Modal Learning (DsCML) to increase the sufficiency of multi-modality information interaction for domain adaptation. For inter-domain cross modal learning, we further advance Cross Modal Adversarial Learning (CMAL) on 2D and 3D data which contains different semantic content aiming to promote high-level modal complementarity. We evaluate our model under various multi-modality domain adaptation settings including day-to-night, country-to-country and dataset-to-dataset, brings large improvements over both uni-modal and multi-modal domain adaptation methods on all settings.
翻译:在面对3D点云上缺少说明时,对域的适应对于成功至关重要。由于3D点云上标记过程耗费了大量时间,因此对3D点点云进行域域的调整是期待很高的。随着多式数据集的崛起,除了3D点云之外,大量2D图像可以进入。鉴于这一点,我们提议进一步利用2D数据,通过跨域跨模式学习来进行3D域的适应。关于跨域内和跨域跨模式学习,大多数现有工作将密集的2D像素特性抽样试样到同一大小,同时具有稀有的3D点点特征,结果放弃了许多有用的 2D 特性。为了解决这个问题,我们提议动态的稀到宽跨式跨模式学习(DSCML),以提高多模式信息互动的充足性。关于跨域跨域跨模式学习,我们进一步推进了2D 和3D 模型学习(CMAL),其中含有不同的语义内容,目的是促进高层次的3D点点特征互补。我们建议了动态的多式跨域设置,包括大型的多式国家适应模式。我们根据多种模式的模型评估了多式的模型,将大型域的模型的模型对大型域环境的调整,包括大型国内的多式的调整。