A popular application of federated learning is using many clients to train a deep neural network, the parameters of which are maintained on a central server. While recent efforts have focused on reducing communication complexity, existing algorithms assume that each participating client is able to download the current and full set of parameters, which may not be a practical assumption depending on the memory constraints of clients such as mobile devices. In this work, we propose a novel algorithm Comfetch, which allows clients to train large networks using compressed versions of the global architecture via Count Sketch, thereby reducing communication and local memory costs. We provide a theoretical convergence guarantee and experimentally demonstrate that it is possible to learn large networks, such as a deep convolutional network and an LSTM, through federated agents training on their sketched counterparts. The resulting global models exhibit competitive test accuracy when compared against the state-of-the-art FetchSGD and the classical FedAvg, both of which require clients to download the full architecture.
翻译:联合会学习的普及应用正在利用许多客户来培训深层神经网络,其参数在中央服务器上得到维护。虽然最近的努力侧重于降低通信复杂性,但现有的算法假设每个参与的客户能够下载目前和完整的参数,而根据移动设备等客户的记忆限制,这可能不是一个实际的假设。在这项工作中,我们提出了一个新的算法Comfetch,允许客户通过Sletch伯爵使用全球架构的压缩版本来培训大型网络,从而减少通信和当地记忆成本。我们提供了理论趋同保证,并实验性地证明,通过对素描对应方的联邦代理人培训,可以学习大型网络,如深层脉动网络和LSTM。所产生的全球模型在与最先进的FetchSGD和经典FedAvg相比显示出竞争性测试的准确性,这两个模型都要求客户下载整个架构。