Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
翻译:计算、数据和算法进步是指导现代机器学习(ML)进步的三个基本因素。在本文中,我们研究最容易量化的因素(计算 ) 的趋势。我们显示,2010年之前,培训计算根据摩尔的法律增长,大约每20个月翻一番。自2010年代初深造以来,培训计算速度加快,大约每6个月翻一番。在2015年底,随着企业开发大型ML模型,在培训计算方面需要10至100倍的更大要求,出现了新的趋势。根据这些观察,我们把计算ML的历史分为三个时代:深造时代、深造时代和大层时代。总体而言,我们的工作突出了培训先进的ML系统快速增长的计算要求。