Population Predictive Checks

Bayesian modeling has become a staple for researchers to articulate assumptions and develop methods tailored for specific data applications. Thanks to recent developments in approximate posterior inference, researchers can easily build, use, and revise complicated Bayesian models for large and rich data. These new abilities, however, bring into focus the problem of model criticism. Researchers need tools to diagnose the fitness of their models, to understand where they fall short, and to guide their revision. In this paper we develop a new method for Bayesian model criticism, the population predictive check (POP-PC). POP-PCs are built on posterior predictive checks (PPCs), a seminal method that checks a model by assessing the posterior predictive distribution on the observed data. However, PPCs use the data twice -- both to calculate the posterior predictive and to evaluate it -- which can lead to overconfident assessments of the quality of a model. POP-PCs, in contrast, compare the posterior predictive distribution to a draw from the population distribution, which in practice is a heldout dataset. We prove this strategy, which blends Bayesian modeling with frequentist assessment, is calibrated, unlike the PPC. Moreover, we demonstrate that calibrating PPC p-values post-hoc does not resolve the "double use of the data" problem. Finally, we study POP-PCs on classical regression and a hierarchical model of text data.