Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge. Instead of training an intact (or dense) PFL model, FedSpa only maintains a fixed number of active parameters throughout training (aka sparse-to-sparse training), which enables users' models to achieve personalization with cheap communication, computation, and memory cost. We theoretically show that the iterates obtained by FedSpa converge to the local minimizer of the formulated SPFL problem at rate of $\mathcal{O}(\frac{1}{\sqrt{T}})$. Comprehensive experiments demonstrate that FedSpa significantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.
翻译:联邦学习(FL)容易受不同分布数据的影响,因为FL的通用全球模型可能无法适应每个用户的不同数据分布。为了解决这个问题,建议个人化FL(PFL)为每个用户制作专门的本地模型。然而,PFL远不成熟,因为现有的PFL解决方案要么对不同的模型结构进行不满意的概括化,要么花费巨大的额外计算和记忆成本。在这项工作中,我们提议采用个人化的稀薄面具(FedSpa),即新的个人化的PFL计划,利用个人化的稀薄面具将边缘的本地模型定制化。FFFLL(PFL)模式不是进行完整(或密集)的(PFLL)模式培训,而是在整个培训中只保留固定数量的积极参数(aks-spart-sprase培训),使用户模式能够以廉价的通信、计算和记忆成本实现个性化。我们从理论上表明,FedSpa获得的碳含量以$/mathcal{O}(frafracrent sqration-cal decal decilal) as pal decal decal decal demodustress axil decildationaldations pildations paldaldational deceal decealdationaldaldaldaldaldations) 和速度,我们大大节省了几种快速计算。