Reconstructing a 3D surface from colonoscopy video is challenging due to illumination and reflectivity variation in the video frame that can cause defective shape predictions. Aiming to overcome this challenge, we utilize the characteristics of surface normal vectors and develop a two-step neural framework that significantly improves the colonoscopy reconstruction quality. The normal-based depth initialization network trained with self-supervised normal consistency loss provides depth map initialization to the normal-depth refinement module, which utilizes the relationship between illumination and surface normals to refine the frame-wise normal and depth predictions recursively. Our framework's depth accuracy performance on phantom colonoscopy data demonstrates the value of exploiting the surface normals in colonoscopy reconstruction, especially on en face views. Due to its low depth error, the prediction result from our framework will require limited post-processing to be clinically applicable for real-time colonoscopy reconstruction.
翻译:从结肠镜影视中重建3D表面具有挑战性,因为视频框架中的光度和反射度变化可能导致有缺陷的形状预测。为了克服这一挑战,我们利用表面正常矢量的特性,并开发一个两步神经框架,大大改进结肠镜重建质量。经过自我监督正常一致性损失训练的正常深度初始化网络为正常深度改进模块提供深度初始化,该模块利用照明与表面正常之间的关系来重新细化框架明智的正常和深度预测。我们框架在幻门结肠镜数据上的深度精确性能展示了利用表层正常值,特别是在面部观察中。由于它的深度差错,我们框架的预测结果将需要有限的后处理,才能在临床上适用于实时结肠镜重建。</s>