Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler-Maruyama discretization of stochastic differential equations (SDEs) driven by either Brownian or Levy stable motion. In this study, we argue that SGN is neither Gaussian nor Levy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded. Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.
翻译:深层学习具有根本重要性。 尽管SGD的简单性, 其效率的清晰度仍具有挑战性。 公约中, SGD的成功取决于培训过程中出现的随机性梯度噪音。 基于这一共识, SGD经常被作为由布朗或利维稳定运动驱动的S&CEDE(SDEs)的Euler- Maruyama离散处理和分析。 在这项研究中,我们争辩说, SGN既不是高萨,也不是利维稳定。相反,在SGN系列中出现的短距离相关性的启发下,我们提议, SGD的成功可以被视为由偏小布朗运动驱动的SDE(SGN)离散。 因此, SGD动态的不同趋同行为是有道理的。 此外, 由FBMM驱动的SDE(SDE)第一次通过时间大概可以推导出。 结果表明, 更大的Hurst参数的逃逸率较低, 并且SGD保持更长的迷你马。 这恰巧与众所周知的SGD(SGD) 偏好地展示了我们对SGD(SGD) 各种记忆结构的模型的模型和模型的模拟实验, 展示了我们各种模型的模型的模型的模型的模型的模型的模型和模型的模型的模型的模型的模型的模拟效果。