Analytics, have some humility: a statistical view of fourth-down decision making

The standard mathematical approach to fourth-down decision making in American football is to make the decision that maximizes estimated win probability. Win probability estimates arise from a statistical model fit from historical data. These machine learning models, however, are overfit high-variance estimators, exacerbated by the highly correlated nature of football play-by-play data. We develop a machine learning framework that accounts for this auto-correlation and knits uncertainty quantification into our decision making. In particular, we recommend a fourth-down decision when we are confident it has higher win probability than all other decisions. Our final product is a major advance in fourth-down strategic decision making: far fewer fourth-down decisions are as obvious as analysts claim.