Advances in Deep Learning have led to the emergence of Large Language Models (LLMs) such as OpenAI Codex which powers GitHub Copilot. LLMs have been fine tuned and packaged so that programmers can use them in an Integrated Development Environment (IDE) to write code. An emerging line of work is assessing the code quality of code written with the help of these LLMs, with security studies warning that LLMs do not fundamentally have any understanding of the code they are writing, so they are more likely to make mistakes that may be exploitable. We thus conducted a user study (N=58) to assess the security of code written by student programmers when guided by LLMs. Half of the students in our study had the help of the LLM and the other half did not. The students were asked to write code in C that performed operations over a singly linked list, including node operations such as inserting, updating, removing, combining, and others. Our study shows that the students who had the help of an LLM were more likely to write functional code, and although security impacts were observed for certain individual functions, no statistically significant impact on security was observed across all functions. We also investigate systematic stylistic differences between unaided and LLM-assisted code, finding that LLM code is more repetitive, which may have an amplifying effect if vulnerable code is repeated in addition to the impact on source code attribution.
翻译:深层学习的进展导致了大型语言模型(LLMs)的出现,如OpenAI Codex(GitHub Copilot权力的GitHub Copilation)。LLMs经过了细微的调整和包装,使程序员能够在综合开发环境中(IDE)使用它们来写代码。正在出现的工作路线是评估在这些LMs的帮助下编写的代码的代码的代码质量,安全研究警告LLMs根本上不理解他们正在编写的代码,因此他们更有可能犯可能被利用的错误。我们因此进行了一项用户研究(N=58),以评估学生程序员在LMs指导下编写的代码的安全性。我们研究中有一半的学生得到了LMM的帮助,而另一半学生没有这样做。我们要求学生在C里写代码代码的代码,在单项链接列表上运行的代码,包括诸如插入、更新、删除、合并等等的操作。我们的研究表明,帮助LM公司添加代码的学生更有可能写出功能代码。虽然在特定个人功能中观察到安全影响,在统计上没有重大影响,但SIM码对Sylreal规则的系统定位可能影响。我们调查了所有功能之间对SylCreal的重复法源的影响。