Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.
翻译:深网络架构在不忘先前的任务的情况下努力不断学习新任务。 最近的一个趋势表明,基于参数扩展的动态架构可以在不断学习的过程中有效减少灾难性的遗忘。 然而,现有办法往往需要在测试时使用任务识别符,需要复杂的调整以平衡不断增加的参数数量,而几乎不共享任何不同任务的信息。 因此,它们难以在没有重大管理的情况下将规模扩大到大量任务。 在本文件中,我们提议了一个基于专用编码器/解码器框架的变压器架构。 关键是, 编码器和解码器可以在所有任务中共享。 通过动态扩展特殊符号, 我们将我们解码器网络的每一个前方都专门用于任务分配。 我们的战略尺度与大量任务相比,由于严格控制参数扩展而导致的记忆和时空管理微不足道。 此外, 这个高效的战略不需要任何超参数调整来控制网络的扩展。 我们的模型在大型图像网络100和图像网络1000 的状态性能上取得了极好的结果, 而同时没有同步的动态框架。