The human brain is capable of learning tasks sequentially mostly without forgetting. However, deep neural networks (DNNs) suffer from catastrophic forgetting when learning one task after another. We address this challenge considering a class-incremental learning scenario where the DNN sees test data without knowing the task from which this data originates. During training, Continual-Prune-and-Select (CP&S) finds a subnetwork within the DNN that is responsible for solving a given task. Then, during inference, CP&S selects the correct subnetwork to make predictions for that task. A new task is learned by training available neuronal connections of the DNN (previously untrained) to create a new subnetwork by pruning, which can include previously trained connections belonging to other subnetwork(s) because it does not update shared connections. This enables to eliminate catastrophic forgetting by creating specialized regions in the DNN that do not conflict with each other while still allowing knowledge transfer across them. The CP&S strategy is implemented with different subnetwork selection strategies, revealing superior performance to state-of-the-art continual learning methods tested on various datasets (CIFAR-100, CUB-200-2011, ImageNet-100 and ImageNet-1000). In particular, CP&S is capable of sequentially learning 10 tasks from ImageNet-1000 keeping an accuracy around 94% with negligible forgetting, a first-of-its-kind result in class-incremental learning. To the best of the authors' knowledge, this represents an improvement in accuracy above 10% when compared to the best alternative method.
翻译:人类大脑有能力在不忘的情况下按顺序学习任务。 然而, 深神经网络( DNN) 在学习一个又一个任务时会遭受灾难性的遗忘。 我们处理这一挑战时, 将考虑到一个等级- 高级学习方案, DNN( 先前未受过训练的) 将测试数据从某项任务出发, 而在培训期间, DNN 在 DNN 内部找到一个子网络, 负责解决某项任务。 然后, 在推断期间, CP&S 选择正确的精度子网络, 来预测这项任务。 通过培训 DNN( 先前未受过训练的) 现有神经网络连接, 来创建一个新的子网络, 即 DNNN( CP & S), 它可以消除灾难性的遗忘, 在 DNNNNN 中创建一个专门区域, 负责解决某项任务, 但仍允许相互冲突; CP&S 战略在不同的子网络选择战略下实施, 将最佳性业绩显示给状态- 级的 DNNNNN( 先前未受过训练的) 神经连接的神经- 学习能力, 其中的C- 100 的C- sal- 学习方法在各种的C- 10- IMLVLA 上测试中, 10- mal- mal- mal- mal- mal- mal- mal- mal- kh- mal- kh 中, 和 mess 10- misal- mal- mal- misal- kleg- sal- sal- kleg- kleg- sal- sal- klegleg- kleg- kleg- kleg- kleg- sal- sal- sal- sald- sal- sal- sal- sal- sal- mal- sal- sal- sal- klegal- klectionsmaldaldal- klegal- mal- sal- sal- sal- mal- mal- sal- sal- mal- sal- mal- mal- mal- mal- sal- sal- sal- mal- mal- mal- sal- m