The traditional framework of federated learning (FL) requires each client to re-train their models in every iteration, making it infeasible for resource-constrained mobile devices to train deep-learning (DL) models. Split learning (SL) provides an alternative by using a centralized server to offload the computation of activations and gradients for a subset of the model but suffers from problems of slow convergence and lower accuracy. In this paper, we implement PFSL, a new framework of distributed split learning where a large number of thin clients perform transfer learning in parallel, starting with a pre-trained DL model without sharing their data or labels with a central server. We implement a lightweight step of personalization of client models to provide high performance for their respective data distributions. Furthermore, we evaluate performance fairness amongst clients under a work fairness constraint for various scenarios of non-i.i.d. data distributions and unequal sample sizes. Our accuracy far exceeds that of current SL algorithms and is very close to that of centralized learning on several real-life benchmarks. It has a very low computation cost compared to FL variants and promises to deliver the full benefits of DL to extremely thin, resource-constrained clients.
翻译:传统的联邦学习(FL)框架要求每个客户端在每次迭代中重新训练模型,这使得资源有限的移动设备难以训练深度学习(DL)模型。分割学习(SL)提供了一种选择,通过使用中央服务器来卸载模型子集的激活和梯度计算,但它存在慢收敛和较低准确性的问题。在本文中,我们实现了PFSL,一种新的分布式分割学习框架,在其中大量的细客户端并行执行迁移学习,从预训练的DL模型开始,而不与中央服务器共享他们的数据或标签。我们实现了一个轻量级的个性化客户端模型的步骤,以为其各自的数据分布提供高性能。此外,我们评估了在非独立同分布数据分布和不平等样本大小的各种场景下,客户之间的公平性表现,并遵守工作公平性约束。在几个实际基准测试中,我们的准确率远远超过当前的SL算法,非常接近于集中式学习。与FL变体相比,它的计算成本非常低,有望向极细、资源有限的客户端提供DL的全部好处。