We present the results of the NLP Community Metasurvey. Run from May to June 2022, the survey elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split almost exactly in half on questions about the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us not only to gain insight on the spectrum of beliefs held by NLP researchers, but also to uncover false sociological beliefs where the community's predictions don't match reality. We find such mismatches on a wide range of issues. Among other results, the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its own belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
翻译:2022年5月至6月,调查引发了对争议问题的看法,包括行业对该领域的影响、对AGI的关切和道德问题。我们的结果将具体数字引向若干争议:例如,答卷人几乎一分为二,质疑人工一般智能的重要性,语言模型是否理解语言,语言结构和感知偏见对于解决NLP问题的必要性。此外,调查提出了元问题,要求答卷人预测调查答复的分布情况。这不仅让我们深入了解NLP研究人员所持有的各种信仰,还让我们发现社区预测与现实不符的虚假的社会学信仰。我们发现,在一系列广泛问题上存在这种不匹配现象。除其他结果外,社区高估了自己对基准的效用和缩小解决现实世界问题的潜力的信念,同时低估了自己对语言结构、感知偏见和跨学科科学的重要性的信念。