Objective: Text mining of clinical notes embedded in electronic medical records is increasingly used to extract patient characteristics otherwise not or only partly available, to assess their association with relevant health outcomes. As manual data labeling needed to develop text mining models is resource intensive, we investigated whether off-the-shelf text mining models developed at external institutions, together with limited within-institution labeled data, could be used to reliably extract study variables to conduct association studies. Materials and Methods: We developed multiple text mining models on different combinations of within-institution and external-institution data to extract social factors from discharge reports of intensive care patients. Subsequently, we assessed the associations between social factors and having a do-not-resuscitate/intubate code. Results: Important differences were found between associations based on manually labeled data compared to text-mined social factors in three out of five cases. Adopting external-institution text mining models using manually labeled within-institution data resulted in models with higher F1-scores, but not in meaningfully different associations. Discussion: While text mining facilitated scaling analyses to larger samples leading to discovering a larger number of associations, the estimates may be unreliable. Confirmation is needed with better text mining models, ideally on a larger manually labeled dataset. Conclusion: The currently used text mining models were not sufficiently accurate to be used reliably in an association study. Model adaptation using within-institution data did not improve the estimates. Further research is needed to set conditions for reliable use of text mining in medical research.
翻译:目标:在电子医疗记录中嵌入的临床说明的文字挖掘越来越多地用于提取患者特征,否则没有或只是部分可用的临床说明,以评估其与相关健康结果的关系。由于开发文本采矿模型所需的人工数据标签是资源密集的,我们调查了外部机构开发的非现成文本采矿模型,加上有限的机构内标签数据,是否可以用于可靠地提取研究关联研究变量。材料和方法:我们开发了关于机构内和外部机构数据不同组合的多种文本采矿模型,以从集中护理病人的出勤报告中提取可靠的社会因素。随后,我们评估了社会因素之间的关联,并制定了一个不重复/内插代码。结果:在5个案例中,根据人工标签数据与文字标记社会因素的关联,发现存在重大差异。采用外部机构内手工标签的采矿模型模型,得出了更高的F1级和外部机构内数据,但并非有意义的不同协会。讨论:虽然文本有助于将样本放大分析,从而在研究中发现更大范围的研究数据,但目前不需要使用更可靠的文本。