GAN is one of the most popular and commonly used neural network models. When the model is large and there is a lot of data, the learning process can be delayed. The standard way out is to use multiple devices. Therefore, the methods of distributed and federated training for GANs are an important question. But from an optimization point of view, GANs are nothing more than a classical saddle-point problem: $\min_x \max_y f(x,y)$. Therefore, this paper focuses on the distributed optimization of the smooth stochastic saddle-point problems using Local SGD. We present a new algorithm specifically for our problem -- Extra Step Local SGD. The obtained theoretical bounds of communication rounds are $\Omega(K^{2/3} M^{1/3})$ in strongly-convex-strongly-concave case and $\Omega(K^{8/9} M^{4/9})$ in convex-concave (here $M$ -- number of functions (nodes) and $K$ - number of iterations).
翻译：GAN是最受欢迎和最常用的神经网络模型之一。 当模型巨大且有大量数据时, 学习过程可以推迟。 标准出路是使用多个设备。 因此, 分布和联合GAN培训的方法是一个重要问题。 但从优化角度看, GAN只是典型的马鞍点问题: $\ min_ x\max_y f( x,y) 美元。 因此, 本文的重点是使用本地 SGD 来分配平滑的沙丘点问题的优化。 我们专门为我们的问题提出了一个新的算法 -- -- 超步骤本地 SGD。 获得的通信回合理论界限是 $\ Omega (K ⁇ 2/3} M ⁇ 1/3} 美元 强力凝结口, 和 $\\ omega (K ⁇ 8/9} M ⁇ 4/9} 等 。 因此, 本文的重点是配置- convex- caveveve (e $M$ -- 函数数 (nodes) 和 $K$ - 迭号) 。
相关内容
微信扫码咨询专知VIP会员