After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. We introduce two procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting COPD risk. We derive "Type I and II" regret bounds, which guarantee the procedures are non-inferior to a static model and competitive with an oracle logistic reviser in terms of the average loss. Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI 0.818-0.938). Online recalibration using BLR and MarBLR improved the aECI, attaining 0.265 (95%CI 0.230-0.300) and 0.241 (95%CI 0.216-0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average AUC (aAUC) from 0.767 (95%CI 0.765-0.769) to 0.800 (95%CI 0.798-0.802) and 0.799 (95%CI 0.797-0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually-refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI 0.913-0.935) and 0.925 (95%CI 0.914-0.935), compared to the static model's aAUC of 0.904 (95%CI 0.892-0.916). Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data. BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.
翻译:在部署临床预测模型后,随后收集的数据可用于微调其预测和适应时间变化。由于模型更新具有超升/装配风险,我们研究有性能保障的在线方法。我们引入了两个程序,以持续重新校正或修改一个基本预测模型:巴伊西亚后勤回归(BLR)和马克夫变量,明确模型分布变化(MarBLR)。我们通过模拟和真实世界研究进行经验性评估,预测COPD风险。我们得出了“Type I和II”的遗憾界限,这保证了程序不易升级/装配风险,我们研究了一个静态模型/装配风险。我们引入了两个程序,在平均损失方面,连续调整或修订一个基础预测模型:Besian后勤回归(BLR)和(a)的平均校正模型(aECI)为0.828(95%CIRRI),在BLR和MarBLR(B-LR)的RM(B- mess mess)之间,B-rassion(950.300)和0.24B(995 C(BR) 分别进行大规模修正和B.