Continual learning-the ability to learn many tasks in sequence-is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catastrophic forgetting, where learning new tasks erases knowledge of earlier tasks. While catastrophic forgetting labels the problem, the theoretical reasons for interference between tasks remain unclear. Here, we attempt to narrow this gap between theory and practice by studying continual learning in the teacher-student setup. We extend previous analytical work on two-layer networks in the teacher-student setup to multiple teachers. Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches. In line with recent work, we find that when tasks depend on similar features, intermediate task similarity leads to greatest forgetting. However, feature similarity is only one way in which tasks may be related. The teacher-student approach allows us to disentangle task similarity at the level of readouts (hidden-to-output weights) and features (input-to-hidden weights). We find a complex interplay between both types of similarity, initial transfer/forgetting rates, maximum transfer/forgetting, and long-term transfer/forgetting. Together, these results help illuminate the diverse factors contributing to catastrophic forgetting.
翻译:持续学习 — 在对人工学习系统至关重要的顺序中学习许多任务的能力。 然而,深层次网络的标准培训方法往往会遭受灾难性的遗忘,因为学习新任务会抹去对早期任务的知识。 虽然灾难性的遗忘会给问题贴上标签,但任务之间干扰的理论原因仍然不清楚。 在这里,我们试图通过在师生设置中研究持续学习来缩小理论与实践之间的差距。 我们把以前对师生设置中的两层网络的分析工作扩大到多位教师。 我们利用每位教师来代表不同的任务,我们调查教师之间的关系如何影响学生在任务转换时所表现出的遗忘和转移的程度。 根据最近的工作,我们发现当任务依赖相似的特点时,中间任务相似之处会导致最大的忘记。 然而,相似之处只是可能与任务相关的一种方式。 师生方法使我们能够在阅读( 隐藏到输出的重量) 和特征( 输入到隐藏的重量) 和特性( 隐藏的重量) 我们发现一种复杂的相互作用, 两种类型的类似率, 初步转移/ 使这些灾难性的结果成为共同的 。