Summary quality assessment metrics have two categories: reference-based and reference-free. Reference-based metrics are theoretically more accurate but are limited by the availability and quality of the human-written references, which are both difficulty to ensure. This inspires the development of reference-free metrics, which are independent from human-written references, in the past few years. However, existing reference-free metrics cannot be both zero-shot and accurate. In this paper, we propose a zero-shot but accurate reference-free approach in a sneaky way: feeding documents, based upon which summaries generated, as references into reference-based metrics. Experimental results show that this zero-shot approach can give us the best-performing reference-free metrics on nearly all aspects on several recently-released datasets, even beating reference-free metrics specifically trained for this task sometimes. We further investigate what reference-based metrics can benefit from such repurposing and whether our additional tweaks help.
翻译:简要质量评估指标分为两类:基于参考和无参考。基于参考的指标在理论上更加准确,但受人类书面参考的可用性和质量限制,两者都难以确保。这激励了过去几年中开发的独立于人类书面参考的无参考指标。然而,现有的无参考质量评估指标不能既零射又准确。在本文中,我们建议以偷偷的方式采用零射、又准确的无参考方法:输入文件,根据这些文件生成摘要,作为基于参考的衡量标准的参考。实验结果显示,这种零射方法可以在最近发布的若干数据集的几乎所有方面为我们提供最佳的无参考指标,甚至击败有时专门为此任务培训的无参考指标。我们进一步调查基于参考的指标可从这种重写中受益,以及我们增加的图章是否有帮助。