Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that: workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.
翻译:最近,错误信息问题已经通过基于众包的方法得到解决:评估声明的真实性,而不是依赖少数专家,利用一群非专家。我们研究在大流行病期间,众包是否是评估真实性的有效和可靠方法,针对的是COVID-19的声明,从而解决与敏感和个人问题有关的信息(误差),与作出判断的时间相比,这既与敏感和个人问题相关,也非常近期。在我们的实验中,人群工人被要求评估声明的真实性,并为评估提供证据。除了表明人群能够准确判断声明的真实性外,我们还报告工人行为、工人之间的协议、综合功能的效果、规模变化以及工人背景和偏见等方面的结果。我们进行长期的纵向研究,与有知觉的和有经验的工人一起多次重新启动任务,从而得出关于一段时间内行为和质量变化的重要见解。我们的结果显示:工人能够检测和客观地分类与COVID-19有关的信息(误差19);众包和专家的判断可以改变和总结工人的行为、总体质量分析的质量;工人的背景和最终显示主要分析结果的源头和结果。