Code-generating large language models translate natural language into code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the users natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users' understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively.
翻译:注意:由于机器翻译的质量有限,以下翻译可能不够准确或恰当:
翻译后的标题:
“它想要我说什么”:弥合终端用户程序员和代码生成大型语言模型之间的抽象差距
翻译后的摘要:
代码生成大型语言模型可以将自然语言翻译为代码。但是,只有少部分自然语言能有效地指导代码生成。对于非专业终端用户程序员来说,学习如何使用这些模型是抽象匹配的挑战。我们在电子表格中的数据分析特定情境下研究了这一挑战。在这种情况下,系统使用 Codex 生成器将用户的自然语言查询映射到 Python 代码,执行代码并显示结果。我们提出了“基于文本的匹配”方法,通过将代码转换回系统化且可预测的自然语言语句,来弥合抽象差距。我们对24名受试者进行了思想吐露的盲对照研究,比较了“基于文本的匹配”和基于先前建立的查询框架原则的非基于文本的匹配。我们发现,基于文本的方法提高了终端用户对代码生成模型范围和能力,以及有效使用该模型需要的语言类型的理解。