The goal of this survey is to present an explanatory review of the approximation properties of deep neural networks. Specifically, we aim at understanding how and why deep neural networks outperform other classical linear and nonlinear approximation methods. This survey consists of three chapters. In Chapter 1 we review the key ideas and concepts underlying deep networks and their compositional nonlinear structure. We formalize the neural network problem by formulating it as an optimization problem when solving regression and classification problems. We briefly discuss the stochastic gradient descent algorithm and the back-propagation formulas used in solving the optimization problem and address a few issues related to the performance of neural networks, including the choice of activation functions, cost functions, overfitting issues, and regularization. In Chapter 2 we shift our focus to the approximation theory of neural networks. We start with an introduction to the concept of density in polynomial approximation and in particular study the Stone-Weierstrass theorem for real-valued continuous functions. Then, within the framework of linear approximation, we review a few classical results on the density and convergence rate of feedforward networks, followed by more recent developments on the complexity of deep networks in approximating Sobolev functions. In Chapter 3, utilizing nonlinear approximation theory, we further elaborate on the power of depth and approximation superiority of deep ReLU networks over other classical methods of nonlinear approximation.
翻译:本次调查的目的是对深神经网络的近似特性进行解释性审查。 具体地说, 我们的目标是了解深神经网络如何和为什么优于其他古典线性和非线性近似方法。 本调查由三章组成。 在第一章中, 我们审查了深网络及其构成非线性结构的关键思想和概念。 我们通过在解决回归和分类问题时将神经网络问题正规化为最优化问题来解决回归和分类问题。 我们简要地讨论了在解决优化问题时使用的随机梯度梯度下移算法和后向分析公式,并解决了与神经网络运行有关的几个问题,包括激活功能的选择、成本功能、过度适应问题和正规化。 在第二章中,我们把重点转向的是神经网络的近似理论。 我们首先介绍了多神经近似中的密度概念,特别是研究真实价值连续功能的石度- Weierstras theormorms。 然后, 在线性近近框架内,我们审查了关于向后向网络的密度和趋一致率的一些古典结果, 包括激活功能的选择、费用功能的选择、 超时, 更近于更近的正统的正统的理论 。