Crowdsourcing is a popular method used to estimate ground-truth labels by collecting noisy labels from workers. In this work, we are motivated by crowdsourcing applications where each worker can exhibit two levels of accuracy depending on a task's type. Applying algorithms designed for the traditional Dawid-Skene model to such a scenario results in performance which is limited by the hard tasks. Therefore, we first extend the model to allow worker accuracy to vary depending on a task's unknown type. Then we propose a spectral method to partition tasks by type. After separating tasks by type, any Dawid-Skene algorithm (i.e., any algorithm designed for the Dawid-Skene model) can be applied independently to each type to infer the truth values. We theoretically prove that when crowdsourced data contain tasks with varying levels of difficulty, our algorithm infers the true labels with higher accuracy than any Dawid-Skene algorithm. Experiments show that our method is effective in practical applications.
翻译:众包是一种常用的方法,通过收集工人的吵闹标签来估计地面真实标签。 在这项工作中,我们受到众包应用程序的驱动,让每个工人能够根据任务类型表现出两种程度的准确性。 将传统的Dawid- Skene模型设计的算法应用到这种情景中,其性能受到艰巨任务的限制。 因此, 我们首先扩展这个模型, 允许工人的准确性根据任务未知类型而变化。 然后我们提出一种光谱方法, 按类型划分任务。 在按类型区分任务后, 任何 Dawid- Skeene 算法( 即为Dawid- Skene 模型设计的任何算法) 都可以独立应用到每种类型中来推断真实值。 我们理论上证明, 当众包数据包含不同难度的任务时, 我们的算法推断出真实标签比任何Dawid- Skene 算法都更精确。 实验显示我们的方法在实际应用中是有效的。