Pre-trained large language models (LLMs) such as OpenAI Codex have shown immense potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, the code produced does not have any correctness guarantees around satisfying user's intent. In fact, it is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. In this paper, we take a first step towards addressing the problem above by proposing the workflow of test-driven user-intent formalization (TDUIF), which leverages lightweight user feedback to jointly (a) formalize the user intent as tests (a partial specification), and (b) generates code that meets the formal user intent. To perform a scalable and large-scale automated evaluation of the algorithms without requiring a user in the loop, we describe how to simulate user interaction with high-fidelity using a reference solution. We also describe and implement alternate implementations of several algorithmic components (including mutating and ranking a set of tests) that can be composed for efficient solutions to the TDUIF problem. We have developed a system TICODER that implements several solutions to TDUIF, and compare their relative effectiveness on the MBPP academic code generation benchmark. Our results are promising with using the OpenAI Codex LLM on MBPP: our best algorithm improves the pass@1 code generation accuracy metric from 48.39% to 70.49% with a single user query, and up to 85.48% with up to 5 user queries. Second, we can generate a non-trivial functional unit test consistent with the user intent within an average of 1.69 user queries for 90.40% of the examples for this dataset.
翻译:诸如 OpenAI Codex 等经过事先训练的大型语言模型(LLMs ) 已经显示出巨大的潜力,通过用非正规自然语言(NL)来生成自然代码,使编码的重要方面自动化。然而,所制作的代码没有满足用户意图的正确性保证。事实上,很难界定正确性的概念,因为自然语言可能含糊不清,缺乏正式的语义。在本文件中,我们迈出了解决上述问题的第一步,提出了测试驱动用户意图正规化(TDUIF)的工作流程,该流程将轻量用户反馈用于联合(a) 将用户意图正式化为测试(部分用户定义) ;以及(b) 生成符合正式用户意图的代码。要对算法进行可缩放和大规模自动评价,而不需要循环中的用户,我们描述如何用一个参考解决方案模拟用户与高纤维化的相互作用。我们还描述并实施了若干算法组成部分(包括突变和分级测试)的替代实施,这些部分可以构成对TDUIFF1问题的高效解决方案(部分用户定义规格规格规格规格规格规格),我们开发了一个系统,而使用一个具有可更新的系统内部数据格式数据模型的系统,用于测试的系统内部测试结果的系统,用于测试结果的系统内部的模型的模型。