One of the challenges in federated learning is the non-independent and identically distributed (non-iid) characteristics between heterogeneous devices, which cause significant differences in local updates and affect the performance of the central server. Although many studies have been proposed to address this challenge, they only focus on local training and aggregation processes to smooth the changes and fail to achieve high performance with deep learning models. Inspired by the phenomenon of neural collapse, we force each client to be optimized toward an optimal global structure for classification. Specifically, we initialize it as a random simplex Equiangular Tight Frame (ETF) and fix it as the unit optimization target of all clients during the local updating. After guaranteeing all clients are learning to converge to the global optimum, we propose to add a global memory vector for each category to remedy the parameter fluctuation caused by the bias of the intra-class condition distribution among clients. Our experimental results show that our method can improve the performance with faster convergence speed on different-size datasets.
翻译:联邦学习面临的挑战之一是异构设备之间存在的非独立同分布(non-iid)特征,这会导致本地更新的显著差异,并影响中央服务器的性能。尽管已经提出许多研究来解决这个挑战,但它们仅关注本地训练和聚合过程以平滑变化,并未在深度学习模型中实现高性能。受神经崩塌现象的启发,我们强制每个客户端优化到用于分类的全局最优结构。具体而言,我们将其初始化为随机的简单xEquangular Tight Frame (ETF),并将其固定为所有客户端本地更新过程的单位优化目标。在确保所有客户端都学习收敛到全局最优后,我们建议为每个类别添加全局内存向量,以修复由客户端之间相对于类别条件的偏差引起的参数波动。我们的实验结果表明,我们的方法可以在不同大小的数据集上提高性能并加快收敛速度。