We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.
翻译:我们从远处发现一个由动力和适应性之间不必要地结合而导致的亚当式优化优化器问题。当动力和适应性参数不匹配时,这种组合导致不稳定和差异。在这项工作中,我们提出了一种方法,即拉普克,它使亚当式方法的动力和适应性脱钩。我们表明,脱钩使超参数具有更大的灵活性,并允许在已签字的梯度方法与适应性梯度方法之间进行直截了当的相互交错。我们实验性地表明,拉普克在各种任务上不断提高亚当的速度和稳定性。我们还将拉普克对康韦克斯问题的遗憾捆绑在一起,并表明我们与亚当的界限因一个关键因素而不同,表明其优势。