Many recent studies have probed status bias in the peer-review process of academic journals and conferences. In this article, we investigated the association between author metadata and area chairs' final decisions (Accept/Reject) using our compiled database of 5,313 borderline submissions to the International Conference on Learning Representations (ICLR) from 2017 to 2022. We carefully defined elements in a cause-and-effect analysis, including the treatment and its timing, pre-treatment variables, potential outcomes and causal null hypothesis of interest, all in the context of study units being textual data and under Neyman and Rubin's potential outcomes (PO) framework. We found some weak evidence that author metadata was associated with articles' final decisions. We also found that, under an additional stability assumption, borderline articles from high-ranking institutions (top-30% or top-20%) were less favored by area chairs compared to their matched counterparts. The results were consistent in two different matched designs (odds ratio = 0.82 [95% CI: 0.67 to 1.00] in a first design and 0.83 [95% CI: 0.64 to 1.07] in a strengthened design). We discussed how to interpret these results in the context of multiple interactions between a study unit and different agents (reviewers and area chairs) in the peer-review system.
翻译:我们利用2017年至2022年向国际学习代表大会(学习代表大会)提交的5 313份边界文件汇编数据库,调查了学术期刊和会议同行审议过程中的地位偏见。在本篇文章中,我们利用2017年至2022年向学习代表大会(学习代表大会)提交的5 313份边界文件的汇编数据库,调查了作者元数据与领域主席最后决定(Accep/Reject)之间的联系。我们仔细界定了因果分析中的要素,包括待遇和时机、预处理变量、潜在结果和因果关系无因果关系假设,所有这些都涉及研究单位为文字数据,在Neyman和Rubin的潜在成果(PO)框架之下。我们发现一些薄弱的证据表明,作者元数据与条款的最后决定相关。我们还发现,在额外的稳定假设下,来自高级别机构(最高为30%或最高为20%)的边界文章与与其对应的对应机构相比不那么,与地区主席相比,区域主席的边界文章较不那么好。结果在两个不同的设计中一致(在首期设计中为0.82 [95% CI:0.67-1.00] 和0.83 [95% CI: CI: CI: CI: CI: CI:0.64-1.07-1.07-1.07-1.07) 在强化设计中, 和多级设计中,我们讨论了机构之间如何解释这些结果。