Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. We propose a method wherein instead of a fixed clipping norm, one clips to a value at a specified quantile of the update norm distribution, where the value at the quantile is itself estimated online, with differential privacy. The method tracks the quantile closely, uses a negligible amount of privacy budget, is compatible with other federated learning technologies such as compression and secure aggregation, and has a straightforward joint DP analysis with DP-FedAvg. Experiments demonstrate that adaptive clipping to the median update norm works well across a range of realistic federated learning tasks, sometimes outperforming even the best fixed clip chosen in hindsight, and without the need to tune any clipping hyperparameter.
翻译:在联合学习(FL)的设置中,培训具有用户一级不同隐私的神经网络的现有方法(例如,DP Federed Averging),在联合学习(FL)的设置中,对每个用户的模型更新的贡献进行约束,将其剪贴到一定的不变值。然而,在任务和学习设置中,对剪贴规范没有良好的先验设置:更新规范的分配取决于模型结构和损失、每个设备的数据数量、客户学习率以及可能的其他参数。我们建议一种方法,在最新标准分布的特定数量中,将一个剪贴到一个数值,即每个用户的模型更新标准分布本身的价值是在线估算的,有差异的隐私。这种方法密切跟踪Quantile,使用微不足道的隐私预算,与压缩和安全聚合等其他联邦学习技术兼容,并与DP-FedAvg 进行直接的DP-FedAvg联合DP分析。 实验表明,适应中位更新规范在一系列现实的联邦化学习任务中非常有效,有时甚至超过在后视镜中选择的最佳固定的剪辑,不需要任何超时。