Federated learning (FL) that enables distributed clients to collaboratively learn a shared statistical model while keeping their training data locally has received great attention recently and can improve privacy and communication efficiency in comparison with traditional centralized machine learning paradigm. However, sensitive information about the training data can still be inferred from model updates shared in FL. Differential privacy (DP) is the state-of-the-art technique to defend against those attacks. The key challenge to achieve DP in FL lies in the adverse impact of DP noise on model accuracy, particularly for deep learning models with large numbers of model parameters. This paper develops a novel differentially-private FL scheme named Fed-SMP that provides client-level DP guarantee while maintaining high model accuracy. To mitigate the impact of privacy protection on model accuracy, Fed-SMP leverages a new technique called Sparsified Model Perturbation (SMP), where local models are sparsified first before being perturbed with additive Gaussian noise. Two sparsification strategies are considered in Fed-SMP: random sparsification and top-$k$ sparsification. We also apply R{\'e}nyi differential privacy to providing a tight analysis for the end-to-end DP guarantee of Fed-SMP and prove the convergence of Fed-SMP with general loss functions. Extensive experiments on real-world datasets are conducted to demonstrate the effectiveness of Fed-SMP in largely improving model accuracy with the same level of DP guarantee and saving communication cost simultaneously.
翻译:使分散客户能够合作学习共享的统计模式,同时将其培训数据保留在本地的联邦学习联合会学习(FL),使分布式客户能够合作学习共享的统计模式,这一点最近受到极大关注,并能够提高隐私和通信效率,与传统的中央机器学习模式相比,这可以提高隐私和通信效率;然而,培训数据的敏感信息仍然可以从FL共享的模型更新中推断出来。 不同隐私(DP)是防范这些袭击的最先进技术。在FL中实现DP的关键挑战在于DP噪音对模型准确性的不利影响,特别是对于具有大量模型参数的深层学习模型模型模型。本文开发了一个新的差异式私人FL计划,名为FD-SMP,提供客户一级的DP-MP保证,同时保持高模型准确性。为了减轻隐私保护对模型准确性的影响,Fed-S的精确性,FDS-MP的精度和最高限性FD-S的精度,我们用FD-S的精确性标准,对FD-S的精度进行精确性标准化,对FD-S的精度的精度进行精确性,对FD-S的精度的精度的精度进行精确性分析。