Large language models (LLMs) have made significant progress in various domains, including healthcare. However, the specialized nature of clinical language understanding tasks presents unique challenges and limitations that warrant further investigation. In this study, we conduct a comprehensive evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within the realm of clinical language understanding tasks. These tasks span a diverse range, including named entity recognition, relation extraction, natural language inference, semantic textual similarity, document classification, and question-answering. We also introduce a novel prompting strategy, self-questioning prompting (SQP), tailored to enhance LLMs' performance by eliciting informative questions and answers pertinent to the clinical scenarios at hand. Our evaluation underscores the significance of task-specific learning strategies and prompting techniques for improving LLMs' effectiveness in healthcare-related tasks. Additionally, our in-depth error analysis on the challenging relation extraction task offers valuable insights into error distribution and potential avenues for improvement using SQP. Our study sheds light on the practical implications of employing LLMs in the specialized domain of healthcare, serving as a foundation for future research and the development of potential applications in healthcare settings.
翻译：大型语言模型（LLMs）在各个领域取得了显著进展，包括医疗保健。然而，临床语言理解任务的专业性质提出了独特的挑战和限制，需要进一步研究。在本研究中，我们对目前最先进的 LLMS（即GPT-3.5、GPT-4和Bard）在临床语言理解任务领域进行了全面评估。这些任务涵盖了各种不同的领域，包括命名实体识别、关系抽取、自然语言推理、语义文本相似度、文档分类和问答。我们还引入了一种新的提示策略，自问自答提示（SQP），旨在提高 LLMs 的性能，通过引出与所处理临床情境相关的信息性问题和答案。我们的评估强调了任务特定的学习策略和提示技巧的重要性，以提高 LLMs 在与医疗保健相关的任务中的有效性。此外，我们对具有挑战性的关系抽取任务进行了深入的错误分析，提供了关于错误分布和可能通过 SQP 改进的有价值见解。我们的研究为 LLMs 在医疗保健方面的应用提供了实践意义，为未来的研究和潜在应用的开发奠定了基础。
相关内容
微信扫码咨询专知VIP会员