Training large neural networks is known to be time-consuming, with the learning duration taking days or even weeks. To address this problem, large-batch optimization was introduced. This approach demonstrated that scaling mini-batch sizes with appropriate learning rate adjustments can speed up the training process by orders of magnitude. While long training time was not typically a major issue for model-free deep offline RL algorithms, recently introduced Q-ensemble methods achieving state-of-the-art performance made this issue more relevant, notably extending the training duration. In this work, we demonstrate how this class of methods can benefit from large-batch optimization, which is commonly overlooked by the deep offline RL community. We show that scaling the mini-batch size and naively adjusting the learning rate allows for (1) a reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time, effectively shortening training duration by 3-4x times on average.
翻译:众所周知,大型神经网络培训耗费时间,学习时间需要数天甚至数周。为了解决这一问题,采用了大批量优化办法。这种方法表明,通过适当调整学习率而扩大小型批量规模,可以按数量级加快培训进程。虽然长期培训时间通常不是没有模型的深离线算法的主要问题,但最近引入的实现最先进业绩的组合方法使得这一问题更加相关,特别是延长了培训时间。在这项工作中,我们展示了这一类方法如何能够受益于大批量优化,而深离线的RL社区通常忽视了这种优化。我们表明,扩大小型批量规模和天真地调整学习率可以(1) 降低批量规模,(2) 加大分配外行动的处罚力度,(3) 改进整合时间,有效地平均将培训时间缩短3-4x倍。