分布式机器学习研究将具有大规模数据量和计算量的任务分布式地部署到多台机器上,其核心思想在于“分而治之”,有效提高了大规模数据计算的速度并节省了开销。

VIP内容

在现代人工智能中,大规模深度学习模型已经成为许多重要互联网业务背后的核心技术,如搜索/广告/推荐系统/CV/NLP。BERT、Vision Transformer、GPT-3和Switch Transformer模型将模型规模扩大到10亿甚至数万个参数,几乎所有学习任务的准确性都得到了显著提高。使用云集群的分布式训练是及时成功地训练此类大规模模型的关键。开发更先进的分布式训练系统和算法既可以降低能源成本,也可以让我们训练更大的模型。此外,开发像联邦学习这样的颠覆性学习模式也至关重要,它不仅可以保护用户的隐私,还可以分担处理前所未有的大数据和模型的负载。这次演讲将主要关注大规模模型的分布式ML系统:云集群的动态分布式训练(https://DistML.ai)和边缘设备的大规模联合学习(https://FedML.ai)。在第一部分中,我将介绍PipeTransformer,这是一种用于分布式训练Transformer模型(BERT和ViT)的自动化弹性管道。在PipeTransformer中,我们设计了自适应的飞冻结算法,可以在训练过程中逐步识别和冻结部分层,并设计了弹性流水线系统,可以动态减少GPU资源来训练剩余的激活层,并在已释放的GPU资源上分叉更多的管道,以扩大数据并行度的宽度。第二部分,我将讨论可扩展的联邦学习,用于在资源受限的边缘设备和FedML生态系统上训练大型模型,其目标是针对CV NLP、GraphNN和IoT等多种AI应用在边缘进行无处不在的分布式训练。

地址: https://www.youtube.com/watch?v=AY7pCYTC8pQ

作者: Chaoyang He,美国洛杉矶南加州大学计算机科学系博士研究生

成为VIP会员查看完整内容
0
20

最新内容

Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

0
0
下载
预览

最新论文

Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

0
0
下载
预览
父主题
Top