The goal of continual learning is to improve the performance of recognition models in learning sequentially arrived data. Although most existing works are established on the premise of learning from scratch, growing efforts have been devoted to incorporating the benefits of pre-training. However, how to adaptively exploit the pre-trained knowledge for each incremental task while maintaining its generalizability remains an open question. In this work, we present an extensive analysis for continual learning on a pre-trained model (CLPM), and attribute the key challenge to a progressive overfitting problem. Observing that selectively reducing the learning rate can almost resolve this issue in the representation layer, we propose a simple but extremely effective approach named Slow Learner with Classifier Alignment (SLCA), which further improves the classification layer by modeling the class-wise distributions and aligning the classification layers in a post-hoc fashion. Across a variety of scenarios, our proposal provides substantial improvements for CLPM (e.g., up to 49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split CUB-200 and Split Cars-196, respectively), and thus outperforms state-of-the-art approaches by a large margin. Based on such a strong baseline, critical factors and promising directions are analyzed in-depth to facilitate subsequent research.
翻译:继续学习的目标是提高识别模式在学习按顺序得出的数据方面的绩效。尽管大多数现有工作是在从零开始学习的前提下建立的,但越来越多的努力致力于将培训前的好处纳入培训前的收益。然而,如何适应地利用预先培训的知识来完成每项增量任务,同时保持其普遍性仍然是一个未决问题。在这项工作中,我们为继续学习预先培训模式(CLM)提供了广泛的分析,并将关键挑战归因于逐步的过度适应问题。我们注意到有选择地降低学习率可以几乎解决代表性层面的这一问题,我们建议一种简单但极为有效的方法,名为“Slow Learker,配有分类调整器”(SLACCA),该方法通过模拟类分配和以后热度方式调整分类层结构,进一步改进分类层。在各种设想中,我们的提案为CLPMM(例如,高达49.76%、50.05%、44.69%和40.16%的关键性挑战。在Slipt CIFAR-100、Sliplead imNet-R、Slidiplate CUB-200和Splissional-196),我们提出了一个非常有效的方法,从而进一步改进了分类化的分类化的分类,从而进一步改进了分类层次层次,从而从一个有希望的大型基础到一个可靠的基础,从而在基础上超越一个有希望的基础。</s>