While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods. This study presents a promising direction for effective knowledge transfer in privacy-preserving collaborative learning.
翻译:联邦学习是一种保护本地数据不被泄露的有前途的协作学习方法,但它仍然容易受到白盒攻击,并且很难适应异构客户端。基于知识蒸馏——一种将知识从教师模型转移到学生模型的有效技术——的联邦蒸馏 (FD) 出现作为一种替代范式,它提供了增强的隐私保证并解决了模型异构的问题。然而,由于本地数据分布的变化以及缺乏训练良好的教师模型,FD 面临着挑战,导致错误且模糊的知识共享,并严重降低了模型性能。为了解决这些问题,本文提出了一种选择性知识共享机制 Selective-FD,它包括客户端选择器和服务器选择器,以精确地确定来自本地和集成预测的知识。根据理论的支持,经验研究表明,我们的方法增强了 FD 框架的泛化能力,并一致优于基线方法。这项研究为隐私保护协作学习中的有效知识转移提供了一个有前途的方向。