Gradient-based algorithms are crucial to modern computer-vision and graphics applications, enabling learning-based optimization and inverse problems. For example, photorealistic differentiable rendering pipelines for color images have been proven highly valuable to applications aiming to map 2D and 3D domains. However, to the best of our knowledge, no effort has been made so far towards extending these gradient-based methods to the generation of depth (2.5D) images, as simulating structured-light depth sensors implies solving complex light transport and stereo-matching problems. In this paper, we introduce a novel end-to-end differentiable simulation pipeline for the generation of realistic 2.5D scans, built on physics-based 3D rendering and custom block-matching algorithms. Each module can be differentiated w.r.t sensor and scene parameters; e.g., to automatically tune the simulation for new devices over some provided scans or to leverage the pipeline as a 3D-to-2.5D transformer within larger computer-vision applications. Applied to the training of deep-learning methods for various depth-based recognition tasks (classification, pose estimation, semantic segmentation), our simulation greatly improves the performance of the resulting models on real scans, thereby demonstrating the fidelity and value of its synthetic depth data compared to previous static simulations and learning-based domain adaptation schemes.
翻译:以梯度为基础的算法对于现代计算机和图形应用至关重要,有利于学习为基础的优化和反向问题。例如,事实证明,光现实化的彩色图像不同化管道对于映射 2D 和 3D 域的应用程序极为宝贵。然而,据我们所知,迄今为止没有努力将这些梯度为基础的方法扩展至深度图像的生成(2.5D ),因为模拟结构光深传感器意味着解决复杂的光传输和立体匹配问题。在本文中,我们引入了一个新的端到端的不同模拟管道,用于生成现实的 2.5D 扫描,建于基于物理的 3D 映像和定制的区块匹配算法。每个模块可以有区别 w.r.t 传感器和场景参数;例如,对一些提供的扫描进行自动调整,或者利用管道作为3D-2.5D 变压器,用于在更大的计算机应用中解决复杂的光量和立体问题。我们应用了为各种深深层识别任务(分类化、配置、定型、定型和定型区图) 的深层次模拟方法的培训,从而改进了我们以往的模拟了真实的静态模型,从而模拟了对模型的模拟了数据进行了真实的模拟。