We introduce the continuized Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradient steps at random times. This continuized variant benefits from the best of the continuous and the discrete frameworks: as a continuous process, one can use differential calculus to analyze convergence and obtain analytical expressions for the parameters; and a discretization of the continuized process can be computed exactly with convergence rates similar to those of Nesterov original acceleration. We show that the discretization has the same structure as Nesterov acceleration, but with random parameters. We provide continuized Nesterov acceleration under deterministic as well as stochastic gradients, with either additive or multiplicative noise. Finally, using our continuized framework and expressing the gossip averaging problem as the stochastic minimization of a certain energy function, we provide the first rigorous acceleration of asynchronous gossip algorithms.
翻译:我们引入了内斯特罗夫加速度的紧凑变体, 即内斯特罗夫加速度的紧凑变体, 其变量由连续的时间参数索引。 两个变体按照直线普通差分方程连续混合, 并在随机时间采取梯度步骤。 这个相联变体从最好的连续和离散框架中获益: 作为一种连续过程, 可以使用不同的微积分来分析趋同, 并获得参数的分析表达方式; 与内斯特罗夫原加速度相似的趋同率可以精确地计算同的连结进程。 我们显示离散化的结构与内斯特罗夫加速率相同, 但有随机参数。 我们在确定性以及随机梯度梯度下提供内斯特罗夫加速度的相联配, 并且有添加或多倍复制性噪声。 最后, 利用我们的contincult 框架, 表达八卦中的问题, 作为某种能量功能的随机最小化最小化最小化, 我们提供了非同步八象算法的首度加速度。