The Effect of Omitted Variables on the Sign of Regression Coefficients

Omitted variables are a common concern in empirical research. We show that "Oster's delta" (Oster 2019), a commonly reported measure of regression coefficient robustness to the presence of omitted variables, does not capture sign changes in the parameter of interest. Specifically, we show that any time this measure is large--suggesting that omitted variables may be unimportant--a much smaller value can actually reverse the sign of the parameter of interest. Relatedly, we show that selection bias adjusted estimands can be extremely sensitive to the choice of the sensitivity parameter. Specifically, researchers commonly compute a bias adjustment under the assumption that Oster's delta equals one. Under the alternative assumption that delta is very close to one, but not exactly equal to one, we show that the bias can instead be arbitrarily large. To address these concerns, we propose a modified measure of robustness that accounts for such sign changes, and discuss best practices for assessing sensitivity to omitted variables. We demonstrate this sign flipping behavior in an empirical application to social capital and the rise of the Nazi party, where we show how it can overturn conclusions about robustness, and how our proposed modifications can be used to regain robustness. We implement our proposed methods in the companion Stata module regsensitivity for easy use in practice.