Differentially Private Learning with Adaptive Clipping

We consider the problem of hyperparameter tuning in training neural networks with user-level differential privacy (DP). Existing approaches for DP training (e.g., DP Federated Averaging) involve bounding the contribution of each user's model update by *clipping* them to a fixed norm. However there is no good *a priori* setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. In this work, we propose a method wherein instead of using a fixed clipping norm, one clips to a value at a specified quantile of the distribution of update norms, where the value at the quantile is itself estimated online, with differential privacy. Experiments demonstrate that adaptive clipping to the median update norm works well across a range of federated learning problems, eliminating the need to tune any clipping hyperparameter.