Increasing neural network depth is a well-known method for improving neural network performance. Modern deep architectures contain multiple mechanisms that allow hundreds or even thousands of layers to train. This work is trying to answer if extending neural network depth may be beneficial in a life-long learning setting. In particular, we propose a novel method based on adding new layers on top of existing ones to enable the forward transfer of knowledge and adapting previously learned representations for new tasks. We utilize a method of determining the most similar tasks for selecting the best location in our network to add new nodes with trainable parameters. This approach allows for creating a tree-like model, where each node is a set of neural network parameters dedicated to a specific task. The proposed method is inspired by Progressive Neural Network (PNN) concept, therefore it is rehearsal-free and benefits from dynamic change of network structure. However, it requires fewer parameters per task than PNN. Experiments on Permuted MNIST and SplitCIFAR show that the proposed algorithm is on par with other continual learning methods. We also perform ablation studies to clarify the contributions of each system part.
翻译:提高神经网络深度是改善神经网络性能的著名方法。现代深层建筑包含多种机制,可以培养数百甚至数千个层次。如果扩展神经网络深度对终身学习环境可能有益,这项工作正在试图回答。特别是,我们提出了一个基于在现有结构之上添加新层的新方法,以便能够向前转移知识,并调整先前学到的表述方式,以适应新的任务。我们使用一种方法确定我们网络中选择最佳位置的最相似的任务,以添加具有可训练参数的新节点。这个方法允许创建一个树形模型,其中每个节点都是专门用于具体任务的神经网络参数。拟议方法受到进步神经网络概念的启发,因此没有彩排,从动态网络结构变化中受益。然而,它要求每个任务参数比PNN少。关于MNIST和SpliteCIFAR的实验显示,提议的算法与其他持续学习方法相匹配。我们还进行对比研究,以澄清每个系统部分的贡献。