Proper Scoring Rules for Missing Value Imputation

Given the prevalence of missing data in modern statistical research, a broad range of methods is available for any given imputation task. How does one choose the `best' method in a given application? The standard approach is to select some observations, set their status to missing, and compare prediction accuracy of the methods under consideration for these observations. Besides having to somewhat artificially mask additional observations, a shortcoming of this approach is that the optimal imputation in this scheme chooses the conditional mean if predictive accuracy is measured with RMSE. In contrast, we would like to rank highest methods that can sample from the true conditional distribution. In this paper, we develop a principled and easy-to-use evaluation method for missing value imputation under the missing completely at random (MCAR) assumption. The approach is applicable for discrete and continuous data and works on incomplete data sets, without having to leave out additional observations for evaluation. Moreover, it favors imputation methods that reproduce the original data distribution. We show empirically on a range of data sets and imputation methods that our score consistently ranks true data high(est) and is able to avoid pitfalls usually associated with performance measures such as RMSE. Finally, we provide an R-package with an implementation of our method.