Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999-2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (>85%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.
翻译:在实际中,许多政策评价研究并不试图控制共同政策的影响,这个问题迄今在方法学文献中没有得到广泛的关注。在这项研究中,我们利用蒙特卡洛模拟评估共同政策对共同使用的统计模型在国家政策评价中的表现的影响。模拟条件表明,共同政策的规模不同,政策颁布日期之间间隔的时间长度也不同。从1999-2016年国家生命统计系统(NVSS)获得的结果数据(每年每100,000人中特定国家类阿片死亡率)并没有试图控制共同政策的影响,因此该问题迄今在50个州没有产生长期的年度州一级数据。当共同政策被忽略时(即从分析模型中省略 ),我们的结果表明,特别是在政策颁布迅速时,存在高度的相对偏差( > 85% ) 。此外,预计1999-2016年国家生命统计系统(NVSS) 多重死亡原因档案中得出的结果数据(每年每100,000人中特定国家类阿片死亡率的比例),因此在18年中产生了长期的州一级年度数据。当共同政策被忽略时(即从分析模型中省略 ),我们的结果显示, 高的相对偏差,特别是当政策在迅速颁布政策时,对于所有不断发生周期政策而言, 也有可能影响。