When the data is distributed across multiple servers, efficient data exchange between the servers (or workers) for solving the distributed learning problem is an important problem and is the focus of this paper. We propose a fast, privacy-aware, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, GADMM, is based on Alternating Direct Method of Multiplier (ADMM) algorithm. The key novelty in GADMM is that each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with lower amount of communication in each exchange. We prove that GADMM converges faster than the centralized batch gradient descent for convex loss functions, and numerically show that it is faster and more communication-efficient than the state-of-the-art communication-efficient centralized algorithms such as the Lazily Aggregated Gradient (LAG), in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under time-varying network topology of the workers.
翻译:当数据在多个服务器上分布时,服务器(或工人)之间为解决分布式学习问题的有效数据交换是一个重要问题,也是本文件的重点。我们提出了一个快速、隐私意识和通信高效的分散化框架,以解决分布式机器学习(DML)问题。提议的算法GADMM基于对调的乘数直接法(ADMM)算法。GADMM中的关键新颖之处是,每个工人只与两个相邻工人交换当地培训的模型,从而在每次交换中训练一个通信量较低的全球模型。我们证明,GADMMM比集中的分批梯度下降功能更快地聚合,在数字上表明,它比最先进的通信高效集中式算法(LAGAG)在合成和真实数据集的线性和后勤性回归任务中更快和更高的通信效率。此外,我们提出了GADMMM(D-GADMM)的变式动态GADMM(DM),并证明它在工人时间变化式网络表层下趋于一致。