使用多镜头系统的开放外科工具分类和手用 (Open surgery tool classification and hand utilization using a multi-camera system)

Purpose: The goal of this work is to use multi-camera video to classify open surgery tools as well as identify which tool is held in each hand. Multi-camera systems help prevent occlusions in open surgery video data. Furthermore, combining multiple views such as a Top-view camera covering the full operative field and a Close-up camera focusing on hand motion and anatomy, may provide a more comprehensive view of the surgical workflow. However, multi-camera data fusion poses a new challenge: a tool may be visible in one camera and not the other. Thus, we defined the global ground truth as the tools being used regardless their visibility. Therefore, tools that are out of the image should be remembered for extensive periods of time while the system responds quickly to changes visible in the video. Methods: Participants (n=48) performed a simulated open bowel repair. A Top-view and a Close-up cameras were used. YOLOv5 was used for tool and hand detection. A high frequency LSTM with a 1 second window at 30 frames per second (fps) and a low frequency LSTM with a 40 second window at 3 fps were used for spatial, temporal, and multi-camera integration. Results: The accuracy and F1 of the six systems were: Top-view (0.88/0.88), Close-up (0.81,0.83), both cameras (0.9/0.9), high fps LSTM (0.92/0.93), low fps LSTM (0.9/0.91), and our final architecture the Multi-camera classifier(0.93/0.94). Conclusion: By combining a system with a high fps and a low fps from the multiple camera array we improved the classification abilities of the global ground truth.

翻译：目的 : 这项工作的目标是使用多相机视频对开放手术工具进行分类, 并辨别每只手持有的工具。多相机系统有助于防止在开放手术视频数据中出现隔离。此外, 多相机系统结合多种观点, 如覆盖整个操作字段的上视图相机和侧重于手动和解剖的近距离相机, 可能会提供更全面的外科工作流程视图。然而, 多相机数据聚合带来了新的挑战: 一个工具可以在一个相机中看到, 而不是另一个。因此, 我们定义全球地面真相为正在使用的工具, 不论它们能见度如何。因此, 多相机系统外的工具应该被长时间记住, 而系统能迅速对视频中的变化作出反应。方法: 参与者( n= 48) 进行了模拟的打开的肠子修理。高视图和近距离相机。 YOLOv5 用于工具和手检。一个高频LSTM, 以每秒30个立方( fps) 和低频LSTM, 以40个第二窗口显示的LS- 090; 将F- RIS 的图像用于空间、高端、高端系统、高端和高端的F- 和F- RIS 的系统。用于空间、高端、高端、高端、高端、高端、高端、低端、低端、低端、低端、低端、低端、低V、低V、LO/ 和低频、低V、LLO、低、低、低、低距离、低、低、低距离、低距离、直、低、低、低距离、低、低、低、低、低、低、低距离、低、低、低距离、低、低、低、低、低、低、低、低、低距离、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、直路、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低、低