Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. Multimodal-GuideNet also provides a visual guidance signal with an error rate of less than 10 pixels for a 224x288 US image.
翻译:在超声波(US)扫描期间,目视跟踪器可以向声学学家提供视觉指导。这种指导对于经验较少的操作者来说具有潜在价值,可以提高他们如何操控探测器的扫描技能,以达到理想的平面。在本文中,建议采用多式指导方法(Multimodal-GuideNet)来捕捉真实的美国视频信号、同步凝视和在一个统一的框架内探测运动之间的分级依赖性。为了理解凝视运动和探测运动之间的因果关系,我们的模型利用多任务学习来联合学习两个相关任务:预测凝视运动和探测有经验的声学家在常规产科扫描中将执行的信号。两种任务都由一种模式-认知空间图联系在一起,以探测多模式投入和分享有用的跨模式信息。Multimodmodal-GuideNet可以通过估计真实扫描的概率分布来扫描多样性。通过三次典型的妇产科扫描检查进行的实验显示,新的方法优于用于探险运动指导的单项任务和视觉运动预测。多式GuideNet还提供比184的图像率低的图像率。