Deep neural networks achieve human-like performance on a variety of perceptual and decision-making tasks. However, networks perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Here, we develop a mathematical and algorithmic framework that enables flexible and continuous training of neural networks on a range of objectives by constructing path connected sets of networks that achieve equivalent functional performance on a given machine learning task. We view the weight space of a neural network as a curved Riemannian manifold and move a network along a functionally invariant path in weight space while searching for networks that satisfy secondary objectives. A path-sampling algorithm trains computer vision and natural language processing networks with millions of weight parameters to learn a series of classification tasks without performance loss while accommodating secondary objectives including network sparsification, incremental task learning, and increased adversarial robustness. Broadly, we conceptualize a neural network as a mathematical object that can be iteratively transformed into distinct configurations by the path-sampling algorithm to define a sub-manifold of networks that can be harnessed to achieve user goals.
翻译:深心神经网络在各种感知和决策任务上达到人性化表现。 然而, 网络在面对变化的任务或目标时表现不佳, 并且基本上无法与人类智能的灵活性和强健性相匹配。 在这里, 我们开发了一个数学和算法框架, 通过构建连接路径的网络组合, 使神经网络能够在特定机器学习任务上实现等效功能性表现, 从而能够灵活和持续地对神经网络进行一系列目标培训。 我们把神经网络的重量空间视为一个曲线的里伊曼元元元, 并将网络在重量空间的功能性变化路径上移动, 并同时寻找满足次级目标的网络。 路径抽样算法将计算机视觉和自然语言处理网络进行路径抽样分析, 用数百万的重量参数来学习一系列分类任务, 而不损失性能, 同时兼顾次级目标, 包括网络的宽度、 递增任务学习, 以及增强对抗性强性强性。 广义上, 我们将神经网络概念化为数学对象, 通过路径抽样算法可以反复转换成不同的组合, 来界定网络的子结构,, 从而实现用户目标。