The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.
翻译:为特定编程问题生成代码解决方案的任务,可以受益于使用诸如可产生多种样本的代码编码x等预先培训语言模型,这种模型可以产生多种不同的样本。然而,这项任务的主要挑战是如何从经过培训的语言模型产生的多个样本中选择最合适的解决方案。一种自然的方法是对照一组测试案例来评估代码解决方案的质量和正确性,但人工生成此类测试案例往往成本高昂且耗时。在本文件中,我们提出了一个新颖的方法,即代码技术,即利用相同的预先培训语言模型自动生成代码样本测试案例,从而减少人类的努力,扩大测试情景的覆盖范围。随后,代码技术中心利用所生成的测试案例执行代码样本,并执行一项双重执行协议,既考虑产出与生成测试案例的一致性,又考虑与其他代码样本的输出结果一致。我们用五种不同的经过培训前语言模型,即代码样本和代码CocoCContests,使用五种不同大小和功能的州前培训前语言模型,从而减少人类的工作,从而扩大测试方案范围。我们的成果显示,CDCT-18-Cont-Cental 将大大改进整个代码选择标准方法,从而实现比以往20 % 标准标准。