We consider the problem of aggregating binary votes from an ensemble of experts to reveal an underlying binary ground truth where each expert votes correctly with some independent probability. We focus on settings where the number of agents is too small for asymptotic results to apply, many experts may vote correctly with low probability, and there is no central authority who knows the experts' competences, or their probabilities of voting correctly. Our approach is to designate a second type of agent -- a judge -- to weight the experts to improve overall accuracy. The catch is that the judge has imperfect competence just like the experts. We demonstrate that having a single minimally competent judge is often better than having none at all. Using an ensemble of judges to weight the experts can provide a better weighting than any single judge; even the optimal weighting under the right conditions. As our results show, the ability of the judge(s) to distinguish between competent and incompetent experts is paramount. Lastly, given a fixed set of agents with unknown competences drawn i.i.d. from a common distribution, we show how the optimal split of the agents between judges and experts depends on the distribution.
翻译:我们考虑从一组专家中汇总二进制选票的问题,以揭示一个基本二进制事实,即每个专家的投票率都正确,且具有某种独立的可能性。我们注重的是,代理人人数太少,无法适用无症状结果的情况,许多专家的投票率可能低,而且没有中央当局知道专家的能力,或他们的投票概率正确。我们的做法是指定第二类代理人 -- -- 一名法官 -- -- 来权衡专家,以提高总体准确性。我们的方法是,指定第二类代理人 -- -- 一名法官 -- -- 来权衡专家,对专家进行权衡,以便提高总体准确性。抓住的是,法官的能力与专家一样,能力不完善。我们证明,仅有一个最能胜任的法官往往胜过一无任何法官。使用一组法官来权衡专家的份量,比任何单一法官都要好;即使是在适当条件下的最佳加权。我们的结果显示,法官区分胜任和不称职的专家的能力至关重要。最后,鉴于从共同分布中抽出一组能力不明的代理人,我们展示了代理人的最佳分工如何取决于法官与专家的分配情况。