Continual learning aims to provide intelligent agents capable of learning multiple tasks sequentially with neural networks. One of its main challenging, catastrophic forgetting, is caused by the neural networks non-optimal ability to learn in non-stationary distributions. In most settings of the current approaches, the agent starts from randomly initialized parameters and is optimized to master the current task regardless of the usefulness of the learned representation for future tasks. Moreover, each of the future tasks uses all the previously learned knowledge although parts of this knowledge might not be helpful for its learning. These cause interference among tasks, especially when the data of previous tasks is not accessible. In this paper, we propose a new method, named Self-Attention Meta-Learner (SAM), which learns a prior knowledge for continual learning that permits learning a sequence of tasks, while avoiding catastrophic forgetting. SAM incorporates an attention mechanism that learns to select the particular relevant representation for each future task. Each task builds a specific representation branch on top of the selected knowledge, avoiding the interference between tasks. We evaluate the proposed method on the Split CIFAR-10/100 and Split MNIST benchmarks in the task agnostic inference. We empirically show that we can achieve a better performance than several state-of-the-art methods for continual learning by building on the top of selected representation learned by SAM. We also show the role of the meta-attention mechanism in boosting informative features corresponding to the input data and identifying the correct target in the task agnostic inference. Finally, we demonstrate that popular existing continual learning methods gain a performance boost when they adopt SAM as a starting point.
翻译:持续学习的目的是提供能够与神经网络相继学习多重任务的知识分子,这种知识分子能够与神经网络相继学习。它的主要挑战性、灾难性的遗忘之一是由神经网络的非最佳性能导致的,在非静止分布中学习。在目前方法的大多数环境中,该媒介从随机初始参数开始,并优化以掌握当前的任务,而不论所学的表述对未来任务的用处如何。此外,今后每一项任务都使用所有先前学到的知识,尽管这种知识的一部分可能不利于其学习。这些知识在任务中造成干扰,特别是在无法获得先前任务的数据时。在本文件中,我们提出一种新的方法,即名为自我-自我意识Meta-Learner(SAM),它学会了一种先前的知识,用于不断学习,从而学习任务序列,同时避免灾难性的遗忘。 SAAM包含一个关注机制,它学会选择与未来每项任务特别相关的代表。每一项任务在选定的知识之上建立一个特定的代表分支,避免任务之间的干扰。我们评价了SFAR-10和Slized MNIST中的拟议方法,在不断推进任务中,我们通过在不断推进的初始化任务中,我们通过学习的高级任务的方法,从而显示我们通过学习的高级的高级任务中的一项高级的成绩,从而显示我们学习的高级的成绩的成绩,从而显示我们在最后的方法,可以显示我们通过某些的成绩的成绩的成绩的升级,从而显示我们学习的成绩的成绩的成绩。