Code review is an effective software quality assurance activity; however, it is labor-intensive and time-consuming. Thus, a number of generation-based automatic code review (ACR) approaches have been proposed recently, which leverage deep learning techniques to automate various activities in the code review process (e.g., code revision generation and review comment generation). We find the previous works carry three main limitations. First, the ACR approaches have been shown to be beneficial in each work, but those methods are not comprehensively compared with each other to show their superiority over their peer ACR approaches. Second, general-purpose pre-trained models such as CodeT5 are proven to be effective in a wide range of Software Engineering (SE) tasks. However, no prior work has investigated the effectiveness of these models in ACR tasks yet. Third, prior works heavily rely on the Exact Match (EM) metric which only focuses on the perfect predictions and ignores the positive progress made by incomplete answers. To fill such a research gap, we conduct a comprehensive study by comparing the effectiveness of recent ACR tools as well as the general-purpose pre-trained models. The results show that a general-purpose pre-trained model CodeT5 can outperform other models in most cases. Specifically, CodeT5 outperforms the prior state-of-the-art by 13.4\%--38.9\% in two code revision generation tasks. In addition, we introduce a new metric namely Edit Progress (EP) to quantify the partial progress made by ACR tools. The results show that the rankings of models for each task could be changed according to whether EM or EP is being utilized. Lastly, we derive several insightful lessons from the experimental results and reveal future research directions for generation-based code review automation.
翻译:代码审查是一种有效的软件质量保证活动;然而,它是一种劳动密集型和耗时的活动;因此,最近提出了若干基于代代自动代码审查(ACR)方法,这些方法利用了深层次的学习技术使代码审查过程中的各种活动自动化(例如代码修订生成和评论生成)。我们发现,先前的作品有三个主要局限性。首先,ACR方法在每项工作中都显示是有益的,但这些方法并不全面相互比较,以表明其优于同行的 ACR 方法。第二,在软件工程(SE)的广泛任务中,SCOT5等通用预培训模式已被证明是有效的。然而,以前的工作还没有对这些模式在代码审查过程中的效能进行调查。第三,以前的工作在很大程度上依赖于Exact Match (EM) 标准,该标准仅侧重于完美的预测,忽视了不完整的答案所取得的积极进展。为了填补这种研究差距,我们进行了一项全面研究,将最近ACRR任务工具的效力与通用预测试模型等通用的预培训模型。 4 最新的工作结果显示的是,我们是如何从GLOF5 之前的模型生成的。</s>