Labelling a large quantity of social media data for the task of supervised machine learning is not only time-consuming but also difficult and expensive. On the other hand, the accuracy of supervised machine learning models is strongly related to the quality of the labelled data on which they train, and automatic sentiment labelling techniques could reduce the time and cost of human labelling. We have compared three automatic sentiment labelling techniques: TextBlob, Vader, and Afinn to assign sentiments to tweets without any human assistance. We compare three scenarios: one uses training and testing datasets with existing ground truth labels; the second experiment uses automatic labels as training and testing datasets; and the third experiment uses three automatic labelling techniques to label the training dataset and uses the ground truth labels for testing. The experiments were evaluated on two Twitter datasets: SemEval-2013 (DS-1) and SemEval-2016 (DS-2). Results show that the Afinn labelling technique obtains the highest accuracy of 80.17% (DS-1) and 80.05% (DS-2) using a BiLSTM deep learning model. These findings imply that automatic text labelling could provide significant benefits, and suggest a feasible alternative to the time and cost of human labelling efforts.
翻译:为受监督的机器学习任务贴上大量社交媒体数据标签不仅耗时,而且困难和昂贵。另一方面,受监督的机器学习模型的准确性与其所培训的标签数据的质量密切相关,自动情绪标签技术可以减少人类标签的时间和成本。我们比较了三种自动情绪标签技术:TextBlob、Vader和Afinn,在没有任何人力帮助的情况下对推文表示情感。我们比较了三种情景:一种是使用培训和测试现有的地面真相标签;第二种是使用自动标签作为培训和测试数据集;而第三个实验则使用三种自动标签技术来标注培训数据集和使用地面真相标签进行测试。对两个推特数据集进行了评估:SemEval-2013(DS-1)和SemEval-2016(DS-2)。结果显示,Afin标签技术获得80.17%(DS-1)和80.05%(DS-2)的最高准确度,使用BILSTM的深度学习模型。这些发现意味着自动文本标签可以提供重大的好处,并提出一种可行的人类时间和成本标签办法。