Recently, deep neural networks (DNNs) have achieved great success in semantically challenging NLP tasks, yet it remains unclear whether DNN models can capture compositional meanings, those aspects of meaning that have been long studied in formal semantics. To investigate this issue, we propose a Systematic Generalization testbed based on Natural language Semantics (SyGNS), whose challenge is to map natural language sentences to multiple forms of scoped meaning representations, designed to account for various semantic phenomena. Using SyGNS, we test whether neural networks can systematically parse sentences involving novel combinations of logical expressions such as quantifiers and negation. Experiments show that Transformer and GRU models can generalize to unseen combinations of quantifiers, negations, and modifiers that are similar to given training instances in form, but not to the others. We also find that the generalization performance to unseen combinations is better when the form of meaning representations is simpler. The data and code for SyGNS are publicly available at https://github.com/verypluming/SyGNS.
翻译:最近,深入的神经网络(DNNS)在抽象地挑战NLP的任务方面取得了巨大的成功,然而,仍然不清楚DNN模型能否捕捉到构成含义,这些含义的方面在正式语义中已经长期研究过。为了调查这一问题,我们建议基于自然语言语义的系统化通用测试台(SyGNS),该测试台的任务是将自然语言的句子映射成多种范围含义表达形式,目的是说明各种语义现象。我们使用SYGNS测试神经网络能否系统地分析包含诸如量化和否定等逻辑表达的新组合的句子。实验显示,变异者和GRU模型可以概括为与形式上的培训实例相似的不可见的定性、否定和修改的组合。我们还发现,当语言表达形式更为简单时,对隐形组合的概括性表现会更好。 SYGNS的数据和代码可在https://github.com/verypluming/SyGNS.上公开查阅。