We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework. However, MURANA also encodes new methods as special cases. We highlight one of them, which we call ELVIRA, and show that it improves upon Loopless SVRG.
翻译:我们建议采用通用差异减少算法,我们称之为Multiple Randomized Algorithm(MURANA),目的是以顺序或分布的方式,最大限度地减少几个顺畅功能和常规功能的总和。我们的方法是与一般的随机操作员共同制定的,这样我们就可以模拟各种战略来降低计算复杂性。例如,MURANA支持通过压缩更新矢量来稀释梯度,并通过压缩更新矢量来减少通信负荷。这种多功能使得MURANA能够在一个统一的框架内覆盖许多现有的随机机制。然而,MURANA还把新方法编码为特殊案例。我们强调其中之一,我们称之为ELVIRA,并表明它在Loopless SVRG上有所改进。