GAN is one of the most popular and commonly used neural network models. When the model is large and there is a lot of data, the learning process can be delayed. The standard way out is to use multiple devices. Therefore, the methods of distributed and federated training for GANs are an important question. But from an optimization point of view, GANs are saddle-point problems: $\min_x \max_y f(x,y)$. Therefore, this paper focuses on the distributed optimization of smooth stochastic saddle-point problems. The first part of the paper is devoted to lower bounds for the distributed methods for saddle-point problems as well as the optimal algorithms by which these bounds are achieved. Next, we present a new algorithm for distributed saddle-point problems - Extra Step Local SGD. In the experimental part of the paper, we use the Local SGD technique in practice. In particular, we train GANs in a distributed manner.
翻译：GAN是最受欢迎和最常用的神经网络模型之一。 当模型巨大且有大量数据时, 学习过程可能会被推迟。 标准的出路是使用多个设备。 因此, 分布式和联合式GAN培训的方法是一个重要问题。 但是, 从优化的角度来说, GAN是马鞍问题: $\ min_ x\ max_y f( x,y) 。 因此, 本文的重点是对平滑的随机轮垫点问题进行分布式优化。 本文的第一部分是用于降低分布式马鞍问题方法的下限, 以及实现这些边框的最佳算法。 接下来, 我们为分布式马鞍点问题提出一种新的算法- 超步骤本地 SGD 。 在本文的实验部分, 我们在实践中使用本地 SGD 技术 。 我们以分布式方式培训 GAN 。
相关内容
微信扫码咨询专知VIP会员