The regression discontinuity (RD) design offers identification of causal effects under weak assumptions, earning it a position as a standard method in modern political science research. But identification does not necessarily imply that causal effects can be estimated accurately with limited data. In this paper, we highlight that estimation under the RD design involves serious statistical challenges and investigate how these challenges manifest themselves in the empirical literature in political science. We collect all RD-based findings published in top political science journals in the period 2009-2018. The distribution of published results exhibits pathological features; estimates tend to bunch just above the conventional level of statistical significance. A reanalysis of all studies with available data suggests that researcher discretion is not a major driver of these features. However, researchers tend to use inappropriate methods for inference, rendering standard errors artificially small. A retrospective power analysis reveals that most of these studies were underpowered to detect all but large effects. The issues we uncover, combined with well-documented selection pressures in academic publishing, cause concern that many published findings using the RD design may be exaggerated.
翻译:回归不连续(RD)设计在薄弱假设下可以确定因果关系,从而在现代政治科学研究中成为标准方法。但识别并不一定意味着根据有限数据可以准确估计因果关系。在本文中,我们强调,RD设计下的估计涉及严重的统计挑战,并调查这些挑战如何在政治科学经验文献中体现出来。我们收集了2009-2018年期间在最高政治科学期刊上发表的基于RD的所有调查结果。公布的结果的分布显示出病理特征;估计往往会比常规的统计意义水平高出很多。用现有数据重新分析所有研究报告表明,研究人员的自由裁量权不是这些特征的主要驱动力。然而,研究人员往往使用不适当的推论方法进行推断,人为地缩小标准错误。回顾力分析表明,大多数这些研究没有足够能力来发现所有但巨大的影响。我们发现的问题,加上学术出版中记录良好的选择压力,令人担心许多使用RD设计发表的研究结果可能会被夸大。</s>