We ascertain and compare the performances of AutoML tools on large, highly imbalanced healthcare datasets. We generated a large dataset using historical administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features types. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application.
翻译:我们利用2019年以前四个不同时间窗口中的历史行政要求,包括人口信息以及疾病编码标志,制作了一个大型数据集;我们随后就这个数据集培训了三个Automal工具,以预测2019年的六种不同的疾病结果,并评估了几个指标的模型性能;Automil工具与基线随机森林模型相比有所改进,但没有明显不同;所有模型在精确召回曲线下记录了低区域,未能预测真实的正数,同时保持了真实的负数;模型性能与流行率没有直接关系;我们提供了一个具体的用法,以说明如何选择一个阈值,在真实和虚假的正数率之间保持最佳平衡,因为这是医疗应用中的一个重要考虑因素;保健数据集对Automil MLML工具提出了若干挑战,包括大样本尺寸、高度失衡和现有特征类型中的局限性;所有模型的可缩放性、实际学习不平衡的复选和组合,以及精确的特征选择,都是实现更好业绩的下一个步骤。我们提供了一个具体的用法例;在三个实验中,可以对具体性能进行预测性标进行预测。