The ability to solve problems is a hallmark of intelligence and has been an enduring goal in AI. AI systems that can create programs as solutions to problems or assist developers in writing programs can increase productivity and make programming more accessible. Recently, pre-trained large language models have shown impressive abilities in generating new codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap rather than actual execution whereas semantic similarity (or equivalence) of two code segments depends only on their ``execution similarity'', i.e., being able to get the same output for a given input.
翻译:解决问题的能力是情报的标志,也是AI的一个持久目标。 AI系统可以创建方案,作为解决问题的办法,或者协助开发者编写程序,可以提高生产力,使编程更加容易。 最近,经过预先培训的大型语言模型在通过自然语言描述、修复错误代码、翻译语言之间的代码和检索相关代码部分来生成新代码方面表现出惊人的能力。然而,对这些模型的评价往往分散在仅仅一项或两项具体任务上,以几种语言、部分颗粒度(例如功能)水平和许多情况下没有适当的培训数据。 更重要的是,在多数情况下,对生成的代码的评价仅以单词法重叠而不是实际执行方式进行,而两个代码部分的语义相似性(或等同性)仅取决于其“执行相似性 ”, 即能够为某项输入获得相同的输出。</s>