Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.
翻译:解决深层学习模式的问题需要研究人员以某种优化方法优化损失函数。 研究界已经开发了一百多个不同的优化功能, 但关于优化各种任务绩效的数据却很少。 特别是, 基准中没有一个基准测试优化者在源代码相关问题上的性能。 然而, 现有基准数据表明, 某些优化者在特定领域可能更有效率。 在这项工作中, 我们测试各种优化者在源代码深度学习模式上的性能, 发现优化者的选择可能对模型质量产生重大影响, 一些业绩较好的优化者之间有高达两倍的得分差异。 我们还发现, RAdam 优化者( 及其与 Lutahehead 信封的修改) 是几乎总是能很好地完成我们所考虑的任务的最佳优化者。 我们的研究结果表明, 需要在代码相关任务中对优化者进行更加广泛的研究, 并表明 ML4SE 社区应该考虑使用RADam 而不是默认的优化者来完成与代码相关的深层学习任务。</s>