AlphaCode is a code generation system for assisting software developers in solving competitive programming problems using natural language problem descriptions. Despite the advantages of the code generating system, the open source community expressed concerns about practicality and data licensing. However, there is no research investigating generated codes in terms of code clone and performance. In this paper, we conduct an empirical study to find code similarities and performance differences between AlphaCode-generated codes and human codes. The results show that (i) the generated codes from AlphaCode are similar to human codes (i.e., the average maximum similarity score is 0.56) and (ii) the generated code performs on par with or worse than the human code in terms of execution time and memory usage. Moreover, AlphaCode tends to generate more similar codes to humans for low-difficulty problems (i.e., four cases have the exact same codes). It also employs excessive nested loops and unnecessary variable declarations for high-difficulty problems, which cause low performance regarding our manual investigation. The replication package is available at https:/doi.org/10.5281/zenodo.6820681
翻译：AlphaCode 是一个代码生成系统,用于协助软件开发者利用自然语言问题描述解决竞争性编程问题。尽管代码生成系统有其优势,开放源代码社区对实用性和数据许可表示担忧。然而,没有从代码克隆和性能方面对生成的代码进行调查研究。在本文件中,我们进行了一项经验研究,以发现阿尔法Code生成的代码与人类代码之间的代码相似和性能差异。结果显示:(一) AlphaCode生成的代码与人类代码相似(即,平均最相似性分为0.56),以及(二)生成的代码在执行时间和记忆使用方面与人类代码相同或更差。此外,对于低难度问题,阿尔法Code往往产生与人类相似的代码(即,四个案例有完全相同的代码)。它也使用过量的嵌套环和不必要变量声明,导致我们手动调查的绩效低。复制软件包见https:/doi.org/10.5281/zeno8281-810606106。
相关内容
微信扫码咨询专知VIP会员