Distributed Mean Estimation (DME) is a fundamental building block in communication efficient federated learning. In DME, clients communicate their lossily compressed gradients to the parameter server, which estimates the average and updates the model. State of the art DME techniques apply either unbiased quantization methods, resulting in large estimation errors, or biased quantization methods, where unbiasing the result requires that the server decodes each gradient individually, which markedly slows the aggregation time. In this paper, we propose QUIC-FL, a DME algorithm that achieves the best of all worlds. QUIC-FL is unbiased, offers fast aggregation time, and is competitive with the most accurate (slow aggregation) DME techniques. To achieve this, we formalize the problem in a novel way that allows us to use standard solvers to design near-optimal unbiased quantization schemes.
翻译:分布式平均估算( DME) 是通信高效联合学习的基本基石。 在 DME 中, 客户向参数服务器传达他们损失的压缩梯度, 该服务器估计平均值并更新模型。 艺术的 DME 技术要么应用不带偏见的量化方法, 导致大量估算错误, 要么使用偏差的量化方法, 其结果要求服务器对每个梯度进行单独解码, 这明显地减缓了聚合时间。 在本文中, 我们提议 QUIC- FL, 这是一种DME 算法, 实现全世界最佳的 。 QUIC- FL 是公正的, 提供快速汇总时间, 并且与最准确的( 慢汇总) DME 技术具有竞争力 。 为了实现这一目标, 我们以一种新颖的方式将问题正式化, 使我们能够使用标准解析器设计近最佳的不偏向性量化计划 。