Current TSA evaluation in a cross-domain setup is restricted to the small set of review domains available in existing datasets. Such an evaluation is limited, and may not reflect true performance on sites like Amazon or Yelp that host diverse reviews from many domains. To address this gap, we present YASO - a new TSA evaluation dataset of open-domain user reviews. YASO contains 2,215 English sentences from dozens of review domains, annotated with target terms and their sentiment. Our analysis verifies the reliability of these annotations, and explores the characteristics of the collected data. Benchmark results using five contemporary TSA systems show there is ample room for improvement on this challenging new dataset. YASO is available at https://github.com/IBM/yaso-tsa.
翻译:目前跨域设置的TSA评价仅限于现有数据集中现有的小型审查领域,这种评价是有限的,可能没有反映从许多领域进行不同审查的亚马逊或叶尔普等地点的真实业绩。为了弥补这一差距,我们介绍了YASO -- -- 开放域用户审查的新的TSA评价数据集。YASO载有来自几十个审查领域的2 215个英文句子,附有目标条款及其情绪。我们的分析核实了这些说明的可靠性,并探讨了所收集的数据的特征。使用当代5个TSA系统的基准结果表明,这一具有挑战性的新数据集有很大的改进余地。 YASO可在https://github.com/IBM/yaso-tsa网站上查阅。