Large language models (LLMs) have recently demonstrated their potential in clinical applications, providing valuable medical knowledge and advice. For example, a large dialog LLM like ChatGPT has successfully passed part of the US medical licensing exam. However, LLMs currently have difficulty processing images, making it challenging to interpret information from medical images, which are rich in information that supports clinical decisions. On the other hand, computer-aided diagnosis (CAD) networks for medical images have seen significant success in the medical field by using advanced deep-learning algorithms to support clinical decision-making. This paper presents a method for integrating LLMs into medical-image CAD networks. The proposed framework uses LLMs to enhance the output of multiple CAD networks, such as diagnosis networks, lesion segmentation networks, and report generation networks, by summarizing and reorganizing the information presented in natural language text format. The goal is to merge the strengths of LLMs' medical domain knowledge and logical reasoning with the vision understanding capability of existing medical-image CAD models to create a more user-friendly and understandable system for patients compared to conventional CAD systems. In the future, LLM's medical knowledge can be also used to improve the performance of vision-based medical-image CAD models.
翻译:大型语言模型(LLMS)最近展示了其在临床应用方面的潜力,提供了宝贵的医疗知识和建议,例如,像ChateGPT这样的大型对话LLM(LLM)成功地通过了美国医学许可考试的一部分内容;然而,LOMS目前难以处理图像,难以解释医学图像中的信息,因为医学图像中的信息丰富,支持临床决策的信息丰富;另一方面,计算机辅助诊断(CAD)医学图像网络在医疗领域取得了巨大成功,使用先进的深层次学习算法支持临床决策;本文介绍了将LMS纳入医疗模拟CAD网络的一种方法;拟议的框架利用LMS加强多种CAD网络的输出,例如诊断网络、损伤分割网络和报告生成网络,通过总结和重组以自然语言文本格式提供的信息;目标是将LADMs的医疗领域知识和逻辑推理与现有医学模拟模型的视觉理解能力结合起来,为病人建立一个比传统的CAD系统更方便用户和理解的系统;今后,LMM的医疗知识模型还可以用来改进以医学为主的CAD。