Federated Learning (FL) is a recent development in the field of machine learning that collaboratively trains models without the training data leaving client devices, in order to preserve data-privacy. In realistic settings, the total training set is distributed over clients in a highly non-Independent and Identically Distributed (non-IID) fashion, which has been shown extensively to harm FL convergence speed and final model performance. We propose a novel, generalised approach for applying adaptive optimisation techniques to FL with the Federated Global Biased Optimiser (FedGBO) algorithm. FedGBO accelerates FL by applying a set of global biased optimiser values during the local training phase of FL, which helps to reduce `client-drift' from non-IID data, whilst also benefiting from adaptive momentum/learning-rate methods. We show that the FedGBO update with a generic optimiser can be viewed as a centralised update with biased gradients and optimiser update, and use this theoretical framework to prove the convergence of FedGBO using momentum-Stochastic Gradient Descent. We also perform extensive experiments using 4 realistic benchmark FL datasets and 3 popular adaptive optimisers to compare the performance of different adaptive-FL approaches, demonstrating that FedGBO has highly competitive performance considering its low communication and computation costs, and providing highly practical insights for the use of adaptive optimisation in FL.
翻译:联邦学习联盟(FL)是机器学习领域的一项最新发展,它通过协作培训模型,而没有培训数据离开客户设备,以保存数据隐私。在现实环境中,全套培训以高度不独立和同样分布(非IID)的方式在客户中分布,这已广泛显示损害FL趋同速度和最后模型性能。我们提出与Fed GBO(FedGBO)(FedGBO)(FedGBO)(FedGBO)(FedGBO)(FedGBO)算法)一道,将适应性优化技术应用于FL(FedGBO)(FedGBO) (FedGBO) (FedGBO) (FedGBO) (FedGBO)(FL) (FedG) (FedGBO) ((FedGBO) (FedG-S) (FD) (FD) (FedG) (FD) (FD) (FedG-S) (FDI) (FI) (不偏向上) (FDI) (FDI-S) (FD) (FDI) (FDI) (FDI) (FDI) (FDI) (FDI) (FDI) (FDI) (FDI) ((FD) (FDI) ((FD) (的) (FD) (FD) (FD) (= ) (FD) (FDI) (FDI) (FDI) (FDI) (FDI) (的) (FDI) (FDI) (的) (的) (的) (的) (的) (FDI) (的) (FD) (FD) (FI) ((FD) ) (= ) ) ) ) ) (FD) ((FD) ((FDI) (FDI) (FD) ((FDI) (FDI) (FD) ((FDI) ((FDI) (=) (=) (=) (=) (=) (FD) (FD) (FD) ) ((FDI) ((FDI) (的)