Precisely assessing the progress in natural language generation (NLG) tasks is challenging, and human evaluation to establish a preference in a model's output over another is often necessary. However, human evaluation is usually costly, difficult to reproduce, and non-reusable. In this paper, we propose a new and simple automatic evaluation method for NLG called Near-Negative Distinction (NND) that repurposes prior human annotations into NND tests. In an NND test, an NLG model must place a higher likelihood on a high-quality output candidate than on a near-negative candidate with a known error. Model performance is established by the number of NND tests a model passes, as well as the distribution over task-specific errors the model fails on. Through experiments on three NLG tasks (question generation, question answering, and summarization), we show that NND achieves a higher correlation with human judgments than standard NLG evaluation metrics. We then illustrate NND evaluation in four practical scenarios, for example performing fine-grain model analysis, or studying model training dynamics. Our findings suggest that NND can give a second life to human annotations and provide low-cost NLG evaluation.
翻译:准确评估自然语言生成(NLG)任务的进展是一项艰巨的任务,因此,人文评价往往必须确定模式产出优于另一种产出。然而,人文评价通常费用昂贵,难以复制,而且不可重复。在本文中,我们提议为NLG(称为NER-NG)任务采用新的简单自动评价方法,将先前的人文说明重新用于NND测试。在NND测试中,NLG模型必须给高质量产出候选人带来更高的可能性,而不是给已知错误的接近负值的候选人带来更大的可能性。通过NND测试一个模型,确定模型的性能,以及模型失败的具体任务错误的分布。通过对NLG三项任务(问题生成、问题回答和总结)的实验,我们表明NND与人文判断的相关性高于标准NLG评价指标。我们然后用四种实际假设来说明NND评价,例如进行精细的模型分析,或研究模型培训动态。我们的研究结果表明NND可以给人以第二个生命,提供低成本的评估。