Federated learning (FL) is a general framework for learning across heterogeneous clients while preserving data privacy, under the orchestration of a central server. FL methods often compute gradients of loss functions purely locally (ie. entirely at each client, or entirely at the server), typically using automatic differentiation (AD) techniques. We propose a federated automatic differentiation (FAD) framework that 1) enables computing derivatives of functions involving client and server computation as well as communication between them and 2) operates in a manner compatible with existing federated technology. In other words, FAD computes derivatives across communication boundaries. We show, in analogy with traditional AD, that FAD may be implemented using various accumulation modes, which introduce distinct computation-communication trade-offs and systems requirements. Further, we show that a broad class of federated computations is closed under these various modes of FAD, implying in particular that if the original computation can be implemented using privacy-preserving primitives, its derivative may be computed using only these same primitives. We then show how FAD can be used to create algorithms that dynamically learn components of the algorithm itself. In particular, we show that FedAvg-style algorithms can exhibit significantly improved performance by using FAD to adjust the server optimization step automatically, or by using FAD to learn weighting schemes for computing weighted averages across clients.
翻译:联邦学习(FL)是一个在中央服务器安排下,在维护数据隐私的同时,在不同客户间学习的通用框架。FL方法通常使用自动区分(AD)技术,纯粹在当地(即每个客户完全,或完全在服务器)计算损失函数的梯度,通常使用自动区分(AD)技术。我们提议了一个联合自动区分(FAD)框架,使包含客户和服务器计算以及它们之间的通信的功能的计算衍生物能够以与现有联邦化技术兼容的方式运行。换句话说,FAD计算跨通信边界的衍生物。我们比照传统的AD,显示FAD可以使用各种累积模式实施损失函数的梯度,这些模式引入了不同的计算-通信交易交易和系统要求。此外,我们表明,在FAD的这些模式下,广泛一类的联运计算方法是封闭的,特别意味着,如果最初的计算方法能够使用隐私-服务器计算,那么其衍生物就只能使用这些原始技术进行计算。我们然后表明,FADD可以使用什么来创建算法,以动态方式大量地学习算算算算算法的组成部分。我们显示,用平均的SBADADADAD系统在FADAD的系统上,我们显示,通过SBADADADADADS压平级平级平平级平级的系统进行自动的升级的系统系统进行升级到FAD。