The Expected Value of Perfect Information for Risk Prediction Models

Risk prediction models are often constructed using a finite development sample, thus the resulting predicted risks are uncertain. The decision-theoretic implications of prediction uncertainty are not sufficiently studied. For risk prediction models, a measure of net benefit can be calculated based on interpreting the positivity threshold as an exchange rate between true and false positive outcomes. Adopting a Bayesian perspective, we apply Value of Information concepts from decision theory to such net benefit calculations when developing a risk prediction model. We define the Expected Value of Perfect Information (EVPI) as the expected gain in net benefit by using the correct predictions as opposed to the proposed model. We suggest bootstrap methods for sampling from the posterior distribution of predictions for EVPI calculation using Monte Carlo simulations. In a case study, we used subsets of data of various sizes from a clinical trial to develop risk prediction models for 30-day mortality after acute myocardial infarction. With sample size of 1,000, EVPI was 0 at threshold values above 0.6, indicating no point in procuring more development data. At thresholds of 0.4-0.6, the proposed model was not net beneficial, but EVPI was positive, indicating that obtaining more development data might be beneficial. Across the entire thresholds, the gain in net benefit by using the correct model was 24% higher than the gain by using the proposed model. EVPI declined with larges sample sizes and was generally low with sample size of 4,000 and above. We summarize an algorithm for incorporating EVPI calculations into the commonly used bootstrap method for optimism correction. Value of Information methods can be applied to explore decision-theoretic consequences of uncertainty in risk prediction, and can complement inferential methods when developing or validating risk prediction models.