Small on-device models have been successfully trained with user-level differential privacy (DP) for next word prediction and image classification tasks in the past. However, existing methods can fail when directly applied to learn embedding models using supervised training data with a large class space. To achieve user-level DP for large image-to-embedding feature extractors, we propose DP-FedEmb, a variant of federated learning algorithms with per-user sensitivity control and noise addition, to train from user-partitioned data centralized in the datacenter. DP-FedEmb combines virtual clients, partial aggregation, private local fine-tuning, and public pretraining to achieve strong privacy utility trade-offs. We apply DP-FedEmb to train image embedding models for faces, landmarks and natural species, and demonstrate its superior utility under same privacy budget on benchmark datasets DigiFace, EMNIST, GLD and iNaturalist. We further illustrate it is possible to achieve strong user-level DP guarantees of $\epsilon<4$ while controlling the utility drop within 5%, when millions of users can participate in training.
翻译:过去已成功地使用用户级差分隐私(DP)在设备上训练小型模型,用于下一个单词预测和图像分类任务。然而,当直接应用现有方法使用大型类空间的监督训练数据学习嵌入模型时,现有方法可能会失败。为了实现用于大型图像到嵌入特征提取器的用户级DP,我们提出了DP-FedEmb,这是一种变种联邦学习算法,具有每个用户灵敏度控制和噪声添加,用于从中央数据中心的用户分区数据进行训练。DP-FedEmb通过虚拟客户端、部分聚合、私有本地微调和公共预训练相结合,实现了强隐私效用权衡。我们将DP-FedEmb应用于训练面部、地标和自然物种的图像嵌入模型,并证明了在基准数据集DigiFace、EMNIST、GLD和iNaturalist上,在相同的隐私预算下具有优越的效用。我们进一步说明,当数百万用户参与培训时,可以实现强用户级DP保证$\epsilon<4$,同时将实用程序下降控制在5%以内。