Deep neural networks achieve human-like performance on a variety of perceptual and decision making tasks. However, deep networks perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Here, we develop a mathematical and algorithmic framework that enables continual training of deep neural networks on a broad range of objectives by defining path connected sets of neural networks that achieve equivalent functional performance on a given machine learning task while modulating network weights to achieve high-performance on a secondary objective. We view the weight space of a neural network as a curved Riemannian manifold and move a neural network along a functionally invariant path in weight space while searching for networks that satisfy a secondary objective. We introduce a path-sampling algorithm that trains networks with millions of weight parameters to learn a series of image classification tasks without performance loss. The algorithm generalizes to accommodate a range of secondary objectives including weight-pruning and weight diversification and exhibits state of the art performance on network compression and adversarial robustness benchmarks. Broadly, we demonstrate how the intrinsic geometry of machine learning problems can be harnessed to construct flexible and robust neural networks.
翻译:深心神经网络在各种感知和决策任务上达到人性相似的性能。 然而,深心网络在面对变化的任务或目标时表现不佳,而且基本上与人类智能的灵活性和强度不相称。在这里,我们开发了一个数学和算法框架,通过界定在特定机器学习任务上实现等效功能的相联神经网络路径组合,从而能够在一系列目标上持续培训深心神经网络,同时调整网络重量以达到高性能的次级目标。我们把神经网络的重量空间看成一个曲线的里伊曼多元,将神经网络移动在重量空间的功能性不变化路径上,同时寻找满足次级目标的网络。我们引入了一种路径抽样算法,用数百万重量参数对网络进行培训,以学习一系列不造成性能损失的图像分类任务。这种算法一般地考虑到一系列次级目标,包括重量调整和重量的多样化,以及显示网络压缩和对抗性强性强性能基准的艺术性能状态。我们广泛地展示机器学习问题固有的几何测量方法可以用来建立灵活和坚固的神经网络。