Pre-trained models (PTMs) have lead to great improvements in natural language generation (NLG). However, it is still unclear how much commonsense knowledge they possess. With the goal of evaluating commonsense knowledge of NLG models, recent work has proposed the problem of generative commonsense reasoning, e.g., to compose a logical sentence given a set of unordered concepts. Existing approaches to this problem hypothesize that PTMs lack sufficient parametric knowledge for this task, which can be overcome by introducing external knowledge or task-specific pre-training objectives. Different from this trend, we argue that PTM's inherent ability for generative commonsense reasoning is underestimated due to the order-agnostic property of its input. In particular, we hypothesize that the order of the input concepts can affect the PTM's ability to utilize its commonsense knowledge. To this end, we propose a pre-ordering approach to elaborately manipulate the order of the given concepts before generation. Experiments show that our approach can outperform the more sophisticated models that have access to a lot of external data and resources.
翻译:预先培训的模型(PTMs)导致自然语言生成的极大改进(NLG),然而,仍然不清楚它们拥有多少常识知识。为了评估NLG模型的常识,最近的工作提出了基因化常识推理问题,例如,根据一套没有顺序的概念,作出一个逻辑的句子。对于这个问题,现有办法假设PTM缺乏足够的参数知识来完成这项任务,而采用外部知识或特定任务的培训前目标可以克服。与这一趋势不同,我们认为,由于PTM投入的顺序-认知属性特性,PTM具有的基因化常识推理固有能力被低估了。特别是,我们假设投入概念的顺序会影响PTM利用其常识知识的能力。为此,我们提议一种预先排序办法,在生成之前详细调整给定概念的顺序。实验表明,我们的方法可以超越能够获取大量外部数据和资源的更复杂的模型。