Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating client. However, in real applications, the devices of clients are usually heterogeneous, and have different computing power. Although big models like BERT have achieved huge success in AI, it is difficult to apply them to heterogeneous FL with weak clients. The straightforward solutions like removing the weak clients or using a small model to fit all clients will lead to some problems, such as under-representation of dropped clients and inferior accuracy due to data loss or limited model representation ability. In this work, we propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities, bigger models for powerful clients and smaller ones for weak clients. We also propose an effective method to share the knowledge among multiple local models with different sizes. In this way, all the clients can participate in the model learning in FL, and the final model can be big and powerful enough. Besides, we propose a momentum knowledge distillation method to better transfer knowledge in big models on powerful clients to the small models on weak clients. Extensive experiments on many real-world benchmark datasets demonstrate the effectiveness of the proposed method in learning accurate models from clients with heterogeneous devices under the FL framework.
翻译:联邦学习(FL)是用隐私保护方式从分散的数据中培训全球模型的一个重要范例。现有的FL方法通常假定全球模型可以针对任何参与客户进行培训。然而,在实际应用中,客户的装置通常是多种多样的,并且具有不同的计算能力。尽管BERT等大型模型在AI中取得了巨大的成功,但很难将其应用于与弱客户的多样化的FL。消除弱客户或使用小型模型来适应所有客户的简单解决方案会导致一些问题,例如,由于数据丢失或模型代表能力有限,客户的下降比例不足和准确性低。在这项工作中,我们提出包容性FL,这是一个包容客户的联邦化学习方法,以解决这一问题。包容性FL的核心想法是将不同规模的模式分配给具有不同计算能力的客户,为强客户设计更大的模型,为弱客户设计较小的模型。我们还提出了一个在多种本地模型中分享知识的有效方法。所有客户都可以参与FL的模型学习,而最终模型则可以足够强大和强大的模型。此外,我们提议在FBRO模型中向强型客户展示一个更强的模型。