Learning rate schedulers have been widely adopted in training deep neural networks. Despite their practical importance, there is a discrepancy between its practice and its theoretical analysis. For instance, it is not known what schedules of SGD achieve best convergence, even for simple problems such as optimizing quadratic objectives. In this paper, we propose Eigencurve, the first family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed. The condition is quite common in practice. Experimental results show that Eigencurve can significantly outperform step decay in image classification tasks on CIFAR-10, especially when the number of epochs is small. Moreover, the theory inspires two simple learning rate schedulers for practical applications that can approximate eigencurve. For some problems, the optimal shape of the proposed schedulers resembles that of cosine decay, which sheds light to the success of cosine decay for such situations. For other situations, the proposed schedulers are superior to cosine decay.
翻译:培养深神经网络时广泛采用了学习率表。 尽管其实践和理论分析之间有着实际重要性的差异, 但实践和理论分析之间存在差异。 例如, 尚不清楚SGD的进度表能取得最佳趋同, 即使是在优化二次目标等简单问题上也是如此。 在本文中, 我们提议, Eigencurve 是SGD在二次目标上的第一个学习率表系列, 能够达到最小最大最佳趋同率( 直至恒定值 ) 。 对于一些问题, 拟议的进度表的优化形状类似于Cosine 衰变, 这在实际中相当常见。 实验结果显示, Eigencurve 大大超过CIFAR- 10 图像分类任务( CIFAR- 10 10 ) 的步衰减速度, 特别是当小于小粒子时。 此外, 理论激励了两种简单的学习率表, 其实际应用可以接近乙金质。 对于一些问题,, 拟议的进度表的形状和Cosine 衰败情况相似, 。