Observational studies are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in observational studies require many untestable assumptions. This lack of verifiability makes it difficult both to compare different observational study methods and to trust the results of any particular observational study. In this work, we propose TrialVerify, a new approach for evaluating observational study methods based on ground truth sourced from clinical trial reports. We process trial reports into a denoised collection of known causal relationships that can then be used to estimate the precision and recall of various observational study methods. We then use TrialVerify to evaluate multiple observational study methods in terms of their ability to identify the known causal relationships from a large national insurance claims dataset. We found that inverse propensity score weighting is an effective approach for accurately reproducing known causal relationships and outperforms other observational study methods. TrialVerify is made freely available for others to evaluate observational study methods.
翻译:观察性研究对于估计各种医疗干预的效果很有价值,但众所周知,很难评估,因为观察性研究使用的方法需要许多无法检验的假设。这种缺乏可核查性使得难以比较不同的观察性研究方法和相信任何特定观察性研究的结果。在这项工作中,我们提议进行试验核查,这是根据临床试验报告中的实地真相来评价观察性研究方法的新方法。我们把试验报告编成一个公开的已知因果关系汇编,然后用来估计各种观察性研究方法的精确度和回顾。然后我们用试验性核查来评价多种观察性研究方法,从它们能够查明大型国家保险索赔数据集中已知的因果关系。我们发现,反偏重度加权是一种有效方法,可以准确地复制已知的因果关系,并超越其他观察性研究方法。我们向他人免费提供试验性核查,以评价观察性研究方法。