Designing an optimal deep neural network for a given task is important and challenging in many machine learning applications. To address this issue, we introduce a self-adaptive algorithm: the adaptive network enhancement (ANE) method, written as loops of the form train, estimate and enhance. Starting with a small two-layer neural network (NN), the step train is to solve the optimization problem at the current NN; the step estimate is to compute a posteriori estimator/indicators using the solution at the current NN; the step enhance is to add new neurons to the current NN. Novel network enhancement strategies based on the computed estimator/indicators are developed in this paper to determine how many new neurons and when a new layer should be added to the current NN. The ANE method provides a natural process for obtaining a good initialization in training the current NN; in addition, we introduce an advanced procedure on how to initialize newly added neurons for a better approximation. We demonstrate that the ANE method can automatically design a nearly minimal NN for learning functions exhibiting sharp transitional layers as well as discontinuous solutions of hyperbolic partial differential equations.
翻译:为特定任务设计一个最佳深层神经网络对于许多机器学习应用程序来说很重要,也是具有挑战性的。为了解决这一问题,我们引入了一种自我适应算法:适应性网络增强(ANE)方法,该方法以形式列车、估计和增强的循环形式写成。从小型双层神经网络(NN)开始,步骤列车是解决当前NN的优化问题;步骤估计是用当前NN的解决方案计算一个后端天线/指标;步骤增强是将新的神经元添加到目前的NNN。本文根据计算估计估计/指标开发了新网络增强战略,以确定有多少新的神经元和何时应该将一个新的层次添加到目前的NNN。ANE方法提供了一个在培训当前NN的过程中获得良好初始化的自然过程;此外,我们引入了如何使用当前NNN的解决方案初始化新添加的神经元的先进程序;我们证明ANE方法可以自动设计一个近乎最低限度的NNE,用于学习显示尖度过渡层的功能,以及作为部分偏差方程式的局部方程式。