GitHub Copilot is an artificial intelligence model for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about the impact it will have on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60\% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.
翻译:GitHub Copilit是自动生成自然语言问题描述源代码的人工智能模型。 自2022年6月, COpilot正式免费向所有学生开放, 作为视觉工作室代码等发展环境的插件。 之前探索OpenAI Codex的工作, 即具有权力的CPilot 的基本模型, 已经表明它在典型的 CS1 问题上表现良好, 从而引起了人们对它如何教授入门性课程的影响的关切。 但是, 很少有人知道Copilot 效果不佳的问题类型, 或学生在解决错误时可能与 Copil 的自然语言互动。 我们通过评估可公开获得的166个编程问题数据集的 Copilot 的性能来探索这些问题。 我们发现, 它在第一次尝试中成功解决了大约一半的这些问题, 并且它只用自然语言改变问题描述, 解决了剩余的60 个问题。 我们说, 这种快速工程在最初失败时会成为人类和 Copilot 之间的标准互动, 是一种可能有用的学习活动, 能够促进计算思维技能, 并有可能改变代码的开发技能的性质。