Federated learning (FL) is increasingly deployed among multiple clients (e.g., mobile devices) to train a shared model over decentralized data. To address the privacy concerns, FL systems need to protect the clients' data from being revealed during training, and also control the data leakage through trained models when exposed to untrusted domains. Distributed differential privacy (DP) offers an appealing solution in this regard as it achieves an informed tradeoff between privacy and utility without a trusted server. However, existing distributed DP mechanisms work impractically in the presence of client dropout, resulting in either poor privacy guarantees or degraded training accuracy. In addition, these mechanisms also suffer from severe efficiency issues with long time-to-accuracy training performance. We present Hyades, a distributed differentially private FL framework that is highly efficient and resilient to client dropout. Specifically, we develop a novel 'add-then-remove' scheme where a required noise level can be enforced in each FL training round even though some sampled clients may drop out in the end; therefore, the privacy budget is consumed carefully even in the presence of client dropout. To boost performance, Hyades runs as a distributed pipeline architecture via encapsulating the communication and computation operations into stages. It automatically divides the global model aggregation into several chunk-aggregation tasks and pipelines them for optimal speedup. Evaluation through large-scale cloud deployment shows that Hyades can efficiently handle client dropout in various realistic FL scenarios, attaining the optimal privacy-utility tradeoff and accelerating the training by up to 2.1$\times$ compared to existing solutions.
翻译:联邦学习(FL)越来越多地在多个客户(如移动设备)中部署,以培训一个共享的分散数据模式; 为解决隐私问题,FL系统需要保护客户的数据在培训期间不被披露,同时在接触不受信任的领域时通过经过培训的模式控制数据泄漏; 分散的差别隐私(DP)在这方面提供了一个具有吸引力的解决方案,因为它在隐私和公用事业之间实现知情的权衡,而没有可靠的服务器; 然而,现有的分布式DP机制在客户辍学的情况下运作不切实际,造成隐私保障差或培训准确性下降; 此外,这些机制还存在严重的效率问题,长期时间到准确的培训业绩。 我们向Hyades展示了分布式的私人FL框架,在接触不受信任的领域时,这种分散式的私人FL框架效率很高,对客户的退出具有弹性。 具体地说,我们开发了一个新型的“附加式”计划,在每次FL培训周期中都可强制实施必要的噪音水平,尽管有些抽样客户可能最终退出; 因此,隐私预算在客户辍学的情况下被仔细消耗。 此外,这些机制也存在严重的效率问题,在客户辍学情况下, 。 为了提升业绩,在升级的升级的升级,在通过管道结构结构结构结构结构结构中进行自动升级。