Pretrained transformer-based models have shown high performance in natural language generation task. However, a new wave of interest has surged: automatic programming language generation. This task consists of translating natural language instructions to a programming code. Despite the fact that well-known pretrained models on language generation have achieved good performance in learning programming languages, effort is still needed in automatic code generation. In this paper, we introduce JaCoText, a model based on Transformers neural network. It aims to generate java source code from natural language text. JaCoText leverages advantages of both natural language and code generation models. More specifically, we study some findings from the state of the art and use them to (1) initialize our model from powerful pretrained models, (2) explore additional pretraining on our java dataset, (3) carry out experiments combining the unimodal and bimodal data in the training, and (4) scale the input and output length during the fine-tuning of the model. Conducted experiments on CONCODE dataset show that JaCoText achieves new state-of-the-art results.
翻译:预训练的基于Transformer的模型在自然语言生成任务中表现出高性能。然而,自动编程语言生成产生了新的热点。这个任务包括将自然语言指令转化为编程代码。尽管在编程语言学习方面,预训练模型在语言生成方面表现出良好的性能,但在自动代码生成方面仍需要努力。在本文中,我们介绍了JaCoText,这是一种基于Transformers神经网络的模型,旨在从自然语言文本生成Java源代码。JaCoText充分利用了自然语言和代码生成模型的优势。更具体地说,我们研究了现有技术的一些发现,利用它们来(1)从强大的预训练模型初始化我们的模型,(2)在我们的Java数据集上进行额外的预训练,(3)在训练中组合单模态和双模态数据进行实验,以及(4)在模型微调期间扩展输入和输出长度。在CONCODE数据集上进行的实验表明,JaCoText实现了新的最佳结果。