As social media becomes a hotbed for the spread of misinformation, the crucial task of rumor detection has witnessed promising advances fostered by open-source benchmark datasets. Despite being widely used, we find that these datasets suffer from spurious correlations, which are ignored by existing studies and lead to severe overestimation of existing rumor detection performance. The spurious correlations stem from three causes: (1) event-based data collection and labeling schemes assign the same veracity label to multiple highly similar posts from the same underlying event; (2) merging multiple data sources spuriously relates source identities to veracity labels; and (3) labeling bias. In this paper, we closely investigate three of the most popular rumor detection benchmark datasets (i.e., Twitter15, Twitter16 and PHEME), and propose event-separated rumor detection as a solution to eliminate spurious cues. Under the event-separated setting, we observe that the accuracy of existing state-of-the-art models drops significantly by over 40%, becoming only comparable to a simple neural classifier. To better address this task, we propose Publisher Style Aggregation (PSA), a generalizable approach that aggregates publisher posting records to learn writing style and veracity stance. Extensive experiments demonstrate that our method outperforms existing baselines in terms of effectiveness, efficiency and generalizability.
翻译:随着社交媒体成为传播错误信息的温床,传闻检测这一关键任务目睹了开放源基准数据集推动的可喜进展。尽管这些数据集被广泛使用,但我们发现,这些数据集存在虚假的相关性,而现有研究对此视而不见,并导致对现有谣言检测性表现的严重高估。 虚假的相关性源于三个原因:(1) 基于事件的数据收集和标签计划将相同的真实标签赋予来自同一基本事件的多个非常相似的站点;(2) 将多个数据源合并,错误地将来源身份与真实性标签联系起来;(3) 标签偏见。在本文中,我们仔细调查了三种最受欢迎的谣言检测基准数据集(即Twitter15、Twitter16和PHEME),并提议以事件分离的谣言检测作为消除虚假暗示的一种解决办法。在事件间隔的设置下,我们观察到,现有状态模型的准确性标定率大幅下降40%以上,仅与简单的神经分类相近。为了更好地应对这项任务,我们提议,我们提议出版品风格分类(PSBAGGGGG), 张贴一个通用的可计量性标准,并展示我们现有的标准化标准,以展示现有标准化的可实现效率。