Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. GPT-4 shows even higher performance on our manual tests. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets.
翻译：翻译后的摘要：
挖掘逻辑推理能力是一个包罗万象的自然语言理解的工作。随着"先进"于推理任务的自然语言生成模型——第四代预训练转换器（GPT-4）的发布，我们渴望了解GPT-4在各种逻辑推理任务中的表现。本文对多个逻辑推理数据集进行分析，包括LogiQA和ReClor这样的流行基准数据集，以及AR-LSAT这样的新数据集。我们将多项选择阅读理解和自然语言推理任务测试在需要逻辑推理的基准测试上。我们进一步构建了一个逻辑推理的越界数据集，以研究ChatGPT和GPT-4的鲁棒性。我们还进行了ChatGPT和GPT-4的性能比较。实验结果表明，在大多数逻辑推理基准测试上，ChatGPT的表现显著优于RoBERTa微调的方法。GPT-4在我们的手动测试中显示出了更高的性能。在基准数据集中，ChatGPT和GPT-4在LogiQA和ReClor这样的知名数据集上表现相对良好。然而，当处理新发布的和越界的数据集时，性能显著下降。逻辑推理对于ChatGPT和GPT-4仍然具有挑战性，特别是对于越界和自然语言推理数据集。
相关内容
微信扫码咨询专知VIP会员