Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geometric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demonstrate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.
翻译:对单眼内心镜进行精确度测深和现场重建,是机器人外科外科外科手术操作的一项基本任务。然而,传统立体匹配采用双筒镜图像来感知深度信息,由于使用单眼内心镜,很难将这种信息转移到以软机器人为基础的外科手术系统。在本文中,我们提出了一个新框架,将机器人动脉学和单眼内心镜图像与深而不受监督的深层次学习结合到一个单一网络,以便进行量测深度估计,然后实现3D对复杂解剖的重建。具体地说,我们首先通过利用明度-敏度单眼内心眼深度估计方法获取外科场景的相对深度图。然后,相应的内心镜根据非线性优化的几何内心和光度再投影结果进行计算。之后,我们开发了一种由深度驱动力驱动的伸缩缩缩影(DSOSO)算法,以便从运动内心电图中提取缩缩度系数,我们用直径和直深度估算结果,我们形成了一个强的计算结果。我们用直径直径直径的内心机的内心机的内心机图,我们用深度估算,我们用直径的深度估算,我们用直立的内心机的内测测测测测测测测测测测测测测测测深的深度方法,我们用的是,我们用的深度方法,我们用到直径的内心机基的深度方法,我们用了。