Most curriculum learning methods require an approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-free curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that does not require sorting the examples by difficulty and is compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on eight datasets from the computer vision (CIFAR-10, CIFAR-100, Tiny ImageNet), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121), recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures, comparing our approach with the conventional training regime. Moreover, we also compare with Curriculum by Smoothing (CBS), a state-of-the-art data-free curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all datasets and models. Furthermore, we significantly surpass CBS in terms of training time (there is no additional cost over the standard training regime for LeRaC).
翻译:大部分课程学习方法都要求用困难的方法对数据样本进行分类,这往往很麻烦。在这项工作中,我们提出一种新的课程学习方法,称为学习率课程(LeRaC),利用神经网络每一层使用不同的学习率,在初始培训时代创建无数据课程。更具体地说,LeRaC将较高的学习率分配给神经层,随着各层离输入距离更远而逐渐降低学习率。在第一次培训迭代期间,学习率以不同的速度增长,直到它们都达到相同的价值。从这个角度出发,神经模型是按常规神经网络的每一层使用不同的学习率,在初始培训时代创建无数据课程。我们从计算机视野(CIFAR-10、CIFAR-100、Tiny图像网)、语言(BoolQBS、QNLI、RTE)和音频系统(ES-50、CREMA-D)进行相同的培训模式。 考虑各种C-CSB-C-C-CRELA 标准培训(我们C-C-C-CLS-C-C-C-C-CSLAD)的连续培训模式,以及不断的系统(我们C-C-C-C-C-C-C-C-C-C-C-C-C-Serviolverdestrual Statestrual Studal Studal Stastrual T tristrual T)的系统,我们C-tal T tral T tristrual Stal Stal Stal 的系统,我们C-tal 和C-trade) 和C-tradexxx 和C-tradestrual 和C-tal Stental Stental Stental 列的学习的系统,我们的系统,我们C-tal AS-deal AS-de AS AS AS AS AS AS AS AS AS AS) 和C-tal 和C-tal tradeal tradeal Stal Stal Stal Stal Stal 和C-tal AS AS 和C-tal tral tral AS) 和C-tal AS AS AS AS AS 和C-tal AS