Peer-reviewed conferences, the main publication venues in CS, rely critically on matching highly qualified reviewers for each paper. Because of the growing scale of these conferences, the tight timelines on which they operate, and a recent surge in explicitly dishonest behavior, there is now no alternative to performing this matching in an automated way. This paper studies a novel reviewer-paper matching approach that was recently deployed in the 35th AAAI Conference on Artificial Intelligence (AAAI 2021), and has since been adopted (wholly or partially) by other conferences including ICML 2022, AAAI 2022, and IJCAI 2022. This approach has three main elements: (1) collecting and processing input data to identify problematic matches and generate reviewer-paper scores; (2) formulating and solving an optimization problem to find good reviewer-paper matchings; and (3) a two-phase reviewing process that shifts reviewing resources away from papers likely to be rejected and towards papers closer to the decision boundary. This paper also describes an evaluation of these innovations based on an extensive post-hoc analysis on real data -- including a comparison with the matching algorithm used in AAAI's previous (2020) iteration -- and supplements this with additional numerical experimentation.
翻译:同行审评会议是CS的主要出版场所,它非常依赖对每份文件的高度合格审评员进行匹配。由于这些会议的规模不断扩大,其运作时间紧迫,而且最近明显不诚实行为激增,因此现在没有其他办法可以自动进行这种匹配。本文研究的是美国AI协会第三十五届人造情报会议(AAAI 2021)最近采用的新颖的审查员文件匹配方法,此后其他会议,包括ICML 2022、AAAI 2022和ICA 2022等会议(全部或部分)也采用了这一方法。 这种方法有三个主要内容:(1) 收集和处理输入数据,以查明问题匹配,并生成审评员文件评分;(2) 制定和解决优化问题,以找到良好的审评员文件匹配;(3) 分两个阶段审查过程,将审查资源从可能被拒绝的文件转移到接近决定界限的文件。本文件还介绍了根据对真实数据的广泛后分析对这些创新的评价,包括与AAI以往(2020年)的更多数字测试和补充数据进行对比。