We introduce a novel statistical framework to study replicability which simultaneously offers overall Type-I error control, an assessment of compatibility and a combined confidence region. The approach is based on a recently proposed reverse-Bayes method for the analysis of replication success. We show how the method can be recalibrated to obtain a family of combination tests for two studies with exact overall Type-I error control. The approach avoids the double dichotomization for significance of the two-trials rule and has larger project power to detect existing effects. It gives rise to a $p$-value function which can be used to compute a confidence region for the underlying true effect. If the effect estimates are compatible, the resulting confidence interval is similar to the meta-analytic one, but in the presence of conflict, the confidence region splits into two disjoint intervals. The proposed approach is applied to data from the Experimental Economics Replication Project.
翻译:我们引入新的统计框架,研究可复制性,同时提供整体的I类错误控制、兼容性评估和综合信任区域,该方法基于最近提议的反贝耶方法,以分析复制成功率;我们展示该方法如何重新校准,以便为两项研究获得一套综合测试,精确的整体类型I错误控制;该方法避免了两审判规则重要性的双重二分法,并具有更大的项目功率来检测现有效应;该方法产生了一个美元价值的功能,可用于计算信任区域的基本真实效果。如果效果估计是兼容的,由此产生的信任区与元分析区相似,但在发生冲突的情况下,信任区分成两个不连贯的间隔。该拟议方法适用于实验经济学复制项目的数据。