For artificial learning systems, continual learning over time from a stream of data is essential. The burgeoning studies on supervised continual learning have achieved great progress, while the study of catastrophic forgetting in unsupervised learning is still blank. Among unsupervised learning methods, self-supervise learning method shows tremendous potential on visual representation without any labeled data at scale. To improve the visual representation of self-supervised learning, larger and more varied data is needed. In the real world, unlabeled data is generated at all times. This circumstance provides a huge advantage for the learning of the self-supervised method. However, in the current paradigm, packing previous data and current data together and training it again is a waste of time and resources. Thus, a continual self-supervised learning method is badly needed. In this paper, we make the first attempt to implement the continual contrastive self-supervised learning by proposing a rehearsal method, which keeps a few exemplars from the previous data. Instead of directly combining saved exemplars with the current data set for training, we leverage self-supervised knowledge distillation to transfer contrastive information among previous data to the current network by mimicking similarity score distribution inferred by the old network over a set of saved exemplars. Moreover, we build an extra sample queue to assist the network to distinguish between previous and current data and prevent mutual interference while learning their own feature representation. Experimental results show that our method performs well on CIFAR100 and ImageNet-Sub. Compared with the baselines, which learning tasks without taking any technique, we improve the image classification top-1 accuracy by 1.60% on CIFAR100, 2.86% on ImageNet-Sub and 1.29% on ImageNet-Full under 10 incremental steps setting.
翻译:对于人工学习系统来说,必须不断从数据流中不断学习。关于受监督的持续学习的快速研究已经取得了巨大的进步,而对于在未经监督的学习中灾难性的遗忘的研究仍然是空白的。在未经监督的学习方法中,自我监督的学习方法显示在视觉表现方面的巨大潜力,而没有标称规模的数据。为了改进自我监督学习的视觉表现,需要更多和更多不同的数据。在现实世界中,任何时候都会生成无标签的图像数据。这种情况为学习自我监督的持续学习方法提供了巨大的优势。然而,在目前的模式中,将先前的数据和当前数据流中,将先前的数据和当前数据流中的数据和当前数据流合并在一起,我们利用了自我监督的自我评估,将先前的数据和当前数据流中,将前一个数据流的自我监督方法转换到前一个数据流中,我们通过前一个数据流的不断对比的模型和前一个数据流中,我们不用自我监督的模型,我们不用自我监督的对当前数据进行自我评估,将前一个数据流和前一个数据流的系统进行对比性分析,然后将一个数据转换到前一个比前一个数据流的模型的系统。