The Power of Prognosis: Improving Covariate Balance Tests with Outcome Information

Scholars frequently use covariate balance tests to test the validity of natural experiments and related designs. Unfortunately, when measured covariates are unrelated to potential outcomes, balance is uninformative about key identification conditions. We show that balance tests can then lead to erroneous conclusions. To build stronger tests, researchers should identify covariates that are jointly predictive of potential outcomes; formally measure and report covariate prognosis; and prioritize the most individually informative variables in tests. Building on prior research on ``prognostic scores," we develop bootstrap balance tests that upweight covariates associated with the outcome. We adapt this approach for regression-discontinuity designs and use simulations to compare weighting methods based on linear regression and more flexible methods, including machine learning. The results show how prognosis weighting can avoid both false negatives and false positives. To illustrate key points, we study empirical examples from a sample of published studies, including an important debate over close elections.