Stochastic gradient methods (SGMs) are predominant approaches for solving stochastic optimization. On smooth nonconvex problems, a few acceleration techniques have been applied to improve the convergence rate of SGMs. However, little exploration has been made on applying a certain acceleration technique to a stochastic subgradient method (SsGM) for nonsmooth nonconvex problems. In addition, few efforts have been made to analyze an (accelerated) SsGM with delayed derivatives. The information delay naturally happens in a distributed system, where computing workers do not coordinate with each other. In this paper, we propose an inertial proximal SsGM for solving nonsmooth nonconvex stochastic optimization problems. The proposed method can have guaranteed convergence even with delayed derivative information in a distributed environment. Convergence rate results are established to three classes of nonconvex problems: weakly-convex nonsmooth problems with a convex regularizer, composite nonconvex problems with a nonsmooth convex regularizer, and smooth nonconvex problems. For each problem class, the convergence rate is $O(1/K^{\frac{1}{2}})$ in the expected value of the gradient norm square, for $K$ iterations. In a distributed environment, the convergence rate of the proposed method will be slowed down by the information delay. Nevertheless, the slow-down effect will decay with the number of iterations for the latter two problem classes. We test the proposed method on three applications. The numerical results clearly demonstrate the advantages of using the inertial-based acceleration. Furthermore, we observe higher parallelization speed-up in asynchronous updates over the synchronous counterpart, though the former uses delayed derivatives. Our source code is released at https://github.com/RPI-OPT/Inertial-SsGM
翻译:沙变梯度方法( SGM ) 是解决沙发优化的主要方法 。 在平滑的非混凝土问题上, 运用了一些加速技术来提高 SGM 的趋同率 。 然而, 在应用某种加速技术来应用某种超慢的亚梯度方法( SSGM ) 来应对非光滑的非混凝土问题 。 此外, 在分析一个( 加速的) 带有延迟衍生物的 SGM 时, 信息延迟自然发生在一个分布式系统中, 计算工人不相互协调。 在本文中, 我们建议采用惯性快速的SsgMLSGM 来解决非moot的非相近率问题 。 但是, 在分布环境中, 即使在延迟的衍生物分解法( SSGGM ) 中, 混凝固率结果被确定为三类 : 软凝固的调和慢化器, 混合的对调和调和器的调和器的调和器问题 。 对于每个问题, 我们的递化法的递化法 的递化法将显示 。