Predictive risk scores are increasingly used to guide clinical or other interventions in complex settings, particularly healthcare. Directly updating a risk score used to guide interventions leads to biased risk estimates. We propose updating using a `holdout set' -- a subset of the population that does not receive risk-score-guided interventions -- to prevent this. Since samples in the holdout set do not benefit from risk predictions, its size must trade off performance of the updated risk score whilst minimising the number of held out samples. We prove that this approach outperforms simple alternatives, and by defining a general loss function describe conditions under which an optimal holdout size (OHS) can be readily identified. We introduce parametric and semi-parametric algorithms for OHS estimation and demonstrate their use on a recent risk score for pre-eclampsia. Based on these results, we argue that a holdout set is a safe, viable and easily implemented means to safely update predictive risk scores.
翻译:预测性风险评分越来越多地被用于指导复杂环境中的临床或其他干预措施,特别是医疗保健。直接更新用于指导干预措施的风险评分会导致偏差风险估计。我们提议使用“锁定值”进行更新,这是一组没有接受风险核心指导干预措施的人口,以防止发生这种情况。由于“锁定”组中的样本不受益于风险预测,因此其规模必须权衡更新风险评分的绩效,同时尽量减少被搁置样本的数量。我们证明,这一方法优于简单的替代方法,并且通过界定一般损失函数,描述可以随时确定最佳阻滞规模(OHS)的条件。我们为OHS估算采用准参数和半参数算法,并展示其在近期风险评分中用于子宫前的风险计值。基于这些结果,我们认为,“锁定值”组是安全更新预测性风险评分的一种安全、可行和易于实施的手段。