Are We There Yet? Big Surveys Significantly Overestimates COVID-19 Vaccination in the US

Accurate surveys are the primary tool for understanding public opinion towards and barriers preventing COVID-19 vaccine uptake. We compare three prominent surveys about vaccination in the US: Delphi-Facebook ($n\approx 250,000$ per week), Census Household Pulse ($n\approx 75,000$), and Axios-Ipsos ($n\approx 1,000$). We find that the two larger surveys are biased compared to the benchmark from the Centers for Disease Control and Prevention (CDC), and that their sample sizes lead to devastating overconfidence in those incorrect estimates. By April 26, 2021, Delphi-Facebook and Census Household Pulse estimated that at least 73% and 69% of US adults had received a first dose of COVID-19 vaccine, which was 16 and 12 percentage points higher, respectively, than the CDC's estimate (57%). Moreover, estimates of vaccine hesitancy disagree significantly between surveys -- we find that these differences cannot be explained entirely by Delphi-Facebook's under-representation of racial minorities and non-college educated adults. These are examples of the Big Data Paradox: when a confidence interval based on a large but biased sample exhibits both a seriously displaced center and a grossly underestimated width, thus leading us (confidently) away from the truth. With sufficient attention to quality control, small surveys like Axios-Ipsos can be far more reliable than large ones. We leverage a recently established data quality identity (Meng, Annals of Applied Statistics, 2018) to quantify sources of the estimation errors and to conduct a scenario analysis for implications on vaccine willingness and hesitancy. Our study quantifies how bias in large samples can lead to overconfidence in incorrect inferences, which is particularly problematic in studies, like those examined here, that inform high-stakes public policy decisions.