Fact verification has attracted a lot of attention in the machine learning and natural language processing communities, as it is one of the key methods for detecting misinformation. Existing large-scale benchmarks for this task have focused mostly on textual sources, i.e. unstructured information, and thus ignored the wealth of information available in structured formats, such as tables. In this paper we introduce a novel dataset and benchmark, Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS), which consists of 87,026 verified claims. Each claim is annotated with evidence in the form of sentences and/or cells from tables in Wikipedia, as well as a label indicating whether this evidence supports, refutes, or does not provide enough information to reach a verdict. Furthermore, we detail our efforts to track and minimize the biases present in the dataset and could be exploited by models, e.g. being able to predict the label without using evidence. Finally, we develop a baseline for verifying claims against text and tables which predicts both the correct evidence and verdict for 18% of the claims.
翻译:事实核查在机器学习和自然语言处理界引起了许多注意,因为这是发现错误信息的关键方法之一,目前大规模核查工作的基准主要集中于文本来源,即非结构化信息,因此忽视了以结构化格式(如表格)提供的大量信息。在本文中,我们引入了一套新的数据集和基准,即《事实提取和核查,超越结构化和结构化信息》(Feverography),由87 026项经核实的索赔组成。每项索赔都附有说明,并附有维基百科表格中以句子和/或细胞形式提供的证据,以及一个标签,表明这些证据是否支持、反驳或没有提供足够的信息以得出判断。此外,我们详细介绍了我们为追踪和尽量减少数据集中的偏差所作的努力,并可以被模型利用,例如,能够在不使用证据的情况下预测标签。最后,我们制定了根据预测18%索赔正确证据和判断结果的文本和表格核实索赔的基线。