Understanding the extent to which the perceptual world can be recovered from language is a fundamental problem in cognitive science. We reformulate this problem as that of distilling psychophysical information from text and show how this can be done by combining large language models (LLMs) with a classic psychophysical method based on similarity judgments. Specifically, we use the prompt auto-completion functionality of GPT3, a state-of-the-art LLM, to elicit similarity scores between stimuli and then apply multidimensional scaling to uncover their underlying psychological space. We test our approach on six perceptual domains and show that the elicited judgments strongly correlate with human data and successfully recover well-known psychophysical structures such as the color wheel and pitch spiral. We also explore meaningful divergences between LLM and human representations. Our work showcases how combining state-of-the-art machine models with well-known cognitive paradigms can shed new light on fundamental questions in perception and language research.
翻译:认识世界从语言中恢复到什么程度,是认知科学的根本问题。我们将此问题重新表述为从文字中提炼精神物理信息的问题,并展示如何通过将大型语言模型(LLMs)与基于相似性判断的经典精神物理方法相结合来做到这一点。具体地说,我们使用GPT3的即时自动完成功能,即最先进的LMM,在刺激性之间产生相似的分数,然后运用多层面的尺度来发现其内在的心理空间。我们测试了我们在六个概念领域的做法,并表明所得出的判断与人类数据密切相关,并成功恢复了众所周知的心理物理结构,如彩色轮和投球螺旋。我们还探讨了LM和人类表现之间的重大差异。我们的工作展示了将最新机器模型与众所周知的认知范式相结合如何在视觉和语言研究中为根本问题提供新的线索。