Strategic behavior is a fundamental problem in a variety of real-world applications that require some form of peer assessment, such as peer grading of assignments, grant proposal review, conference peer review, and peer assessment of employees. Since an individual's own work is in competition with the submissions they are evaluating, they may provide dishonest evaluations to increase the relative standing of their own submission. This issue is typically addressed by partitioning the individuals and assigning them to evaluate the work of only those from different subsets. Although this method ensures strategyproofness, each submission may require a different type of expertise for effective evaluation. In this paper, we focus on finding an assignment of evaluators to submissions that maximizes assigned expertise subject to the constraint of strategyproofness. We analyze the price of strategyproofness: that is, the amount of compromise on the assignment quality required in order to get strategyproofness. We establish several polynomial-time algorithms for strategyproof assignment along with assignment-quality guarantees. Finally, we evaluate the methods on a dataset from conference peer review.
翻译:战略行为是各种现实世界应用中的一个根本问题,这些应用需要某种形式的同侪评估,例如同侪分配、赠款建议审查、同侪审查和雇员同侪评估。由于个人自己的工作与其评价的提交材料竞争,他们可能提供不诚实的评价,以提高自己提交材料的相对地位。这个问题通常通过个人分割和指派他们只评价不同子群的工作来加以解决。虽然这种方法可以确保战略的可靠性,但每份提交材料可能需要不同种类的专业知识才能进行有效评价。在本文件中,我们着重寻找评价人员来提交材料,以最大限度地利用受战略约束的指定专门知识。我们分析了战略的可靠性:这就是,为了取得战略的可靠性,需要对分配质量作出多大程度的妥协。我们为不受战略限制的任务分配和任务质量保障建立了几个多时算法。最后,我们评估了会议同侪审查的数据集上的方法。