A characteristic feature of human semantic cognition is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties (can breathe) from superordinate concepts (animal) to their subordinates (dog) -- i.e. demonstrate property inheritance. In this paper, we present COMPS, a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties to concepts and their ability to demonstrate property inheritance behavior. Analyses of 22 different PLMs on COMPS reveal that they can easily distinguish between concepts on the basis of a property when they are trivially different, but find it relatively difficult when concepts are related on the basis of nuanced knowledge representations. Furthermore, we find that PLMs can demonstrate behavior consistent with property inheritance to a great extent, but fail in the presence of distracting information, which decreases the performance of many models, sometimes even below chance. This lack of robustness in demonstrating simple reasoning raises important questions about PLMs' capacity to make correct inferences even when they appear to possess the prerequisite knowledge.
翻译:人类语义认知的一个特征是它不仅能够储存和检索通过经验观察到的概念的属性,而且能够便利财产的继承(呼吸)从超大概念(动物)到其下属(狗) -- -- 即展示财产继承。在本文中,我们介绍了一套最低限度的配对句,共同测试预先训练的语言模型(PLMs)将其属性归属于概念的能力及其显示财产继承行为的能力。对22个不同的PLMs在COMS上的分析表明,当概念略有不同时,它们可以很容易地区分以财产为基础的概念,但当概念根据细微知识的表述而相互关联时,它们会发现这些概念相对困难。此外,我们认为,PLMs可以在很大程度上证明与财产继承相一致的行为,但是由于存在分散注意力的信息,这降低了许多模型的性能,有时甚至低于机会。这种缺乏稳健的论证简单推理的能力,引起了关于PLMs是否有能力作出正确的推理的重要问题,即使他们似乎具备必要的知识。