Distributed learning has become a hot research topic, due to its wide application in cluster-based large-scale learning, federated learning, edge computing and so on. Most distributed learning methods assume no error and attack on the workers. However, many unexpected cases, such as communication error and even malicious attack, may happen in real applications. Hence, Byzantine learning (BL), which refers to distributed learning with attack or error, has recently attracted much attention. Most existing BL methods are synchronous, which will result in slow convergence when there exist heterogeneous workers. Furthermore, in some applications like federated learning and edge computing, synchronization cannot even be performed most of the time due to the online workers (clients or edge servers). Hence, asynchronous BL (ABL) is more general and practical than synchronous BL (SBL). To the best of our knowledge, there exist only two ABL methods. One of them cannot resist malicious attack. The other needs to store some training instances on the server, which has the privacy leak problem. In this paper, we propose a novel method, called buffered asynchronous stochastic gradient descent (BASGD), for BL. BASGD is an asynchronous method. Furthermore, BASGD has no need to store any training instances on the server, and hence can preserve privacy in ABL. BASGD is theoretically proved to have the ability of resisting against error and malicious attack. Moreover, BASGD has a similar theoretical convergence rate to that of vanilla asynchronous SGD (ASGD), with an extra constant variance. Empirical results show that BASGD can significantly outperform vanilla ASGD and other ABL baselines, when there exists error or attack on workers.
翻译:分布式学习已成为一个热门的研究课题, 因为它在基于集群的大规模学习、 联合学习、 边缘计算等中广泛应用。 大多数分布式学习方法都假定没有错误和攻击工人。 然而, 许多意外案例, 如通信错误甚至恶意攻击, 可能发生在真实应用中。 因此, 拜占庭学习( BL) 指的是以攻击或错误的方式分散学习, 最近引起了很大的注意。 多数现有的BL 方法是同步的, 当存在混杂工人时, 这将导致缓慢的趋同。 此外, 在一些应用中, 诸如联合学习和边缘计算, 同步甚至无法在网上工人( 客户或边缘服务器服务器) 的多数时间里进行。 因此, 互不协调的 BL (ABL) 可能比同步的 BL (SB) (SB) 更一般和实用。 在我们的所知中, 只有两种ABL 方法。 其中一种是无法抵抗恶意攻击。 当服务器存在隐私泄漏问题时, 另一种需要将一些培训案例存放在服务器上。 在本文中, 我们提议一种新的方法, 将缓冲式方法称为缓冲式的BGASSARC 变变变变变变变变变变变变变 。