Improving automatically generated code from Codex via Automated Program Repair

Large language models, e.g., Codex and AlphaCode, have shown capability in producing working code for many programming tasks. However, the success rate of existing models remains low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics (e.g., type information), resulting in incorrect programs (or even programs which do not compile). In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance confidence in the code produced by language models. Our study revealed that: (1) automatically generated codes share some common programming mistakes with human-crafted solutions, indicating existing APR tools have the potential to fix auto-generated code; (2) TBar and Recoder, two well-known Java APR tools based on templates and learning respectively, increase the number of solved tasks from 37 to 42 on 60 easy level tasks, while increase from 5 to 9 on 53 medium-level programming tasks; (3) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports changing existing code, may outperform existing APR tools in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions on how to improve current APR tools.