Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Due to rapid technological advances and their extreme versatility, LLMs nowadays have millions of users and are at the cusp of being the main go-to technology for information retrieval, content generation, problem-solving, etc. Therefore, it is of great importance to thoroughly assess and scrutinize their capabilities. Due to increasingly complex and novel behavioral patterns in current LLMs, this can be done by treating them as participants in psychology experiments that were originally designed to test humans. For this purpose, the paper introduces a new field of research called "machine psychology". The paper outlines how different subfields of psychology can inform behavioral tests for LLMs. It defines methodological standards for machine psychology research, especially by focusing on policies for prompt designs. Additionally, it describes how behavioral patterns discovered in LLMs are to be interpreted. In sum, machine psychology aims to discover emergent abilities in LLMs that cannot be detected by most traditional natural language processing benchmarks.
翻译:大型语言模型(LLMs)是当前将人工智能系统与人类通信和日常生活紧密结合的前沿技术。由于技术快速发展和其极高的适用性,LLMs现在拥有数百万用户,并且正在成为信息检索、内容生成、问题解决等的主要技术。因此,彻底评估和审查其能力非常重要。由于当前LLMs中日益复杂和新颖的行为模式,可以通过将它们作为人类参与心理学实验的对象来进行评估和审查。因此,本文介绍了一种名为“机器心理学”的新领域。本文概述了心理学不同子领域如何为LLMs定义行为测试,并制定机器心理学研究的方法论标准,特别是重点关注快速设计的策略。此外,本文还描述了如何解释LLMs中发现的行为模式。总之,机器心理学旨在发现LLMs中无法通过大多数传统自然语言处理基准测试检测到的新的出现的能力。