One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activations including most of existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is: {\em any nAI activation function is universal}. It turns out that most of existing activations are nAI, and thus universal in the space of continuous functions on compacta. The framework has the following main properties. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is the first unified attempt that is valid for most of existing activations. Third, as a by product, the framework provides the first university proof for some of the existing activation functions including Mish, SiLU, ELU, GELU, and etc. Fourth, it provides new proofs for most activation functions. Fifth, it discovers new activations with guaranteed universality property. Sixth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with predetermined number of neurons, and the values of weights/biases. Seventh, the framework allows us to abstractly present the first universal approximation with favorable non-asymptotic rate.
翻译:许多神经网络能够复制复杂任务或功能的原因之一是其普遍性特性。虽然过去几十年在神经网络理论方面取得了巨大的进步,但神经网络普遍性的单一建设性框架仍然缺乏。本文件旨在提供一个统一和建设性的框架,以便实现包括大多数现有网络在内的大规模激活的普遍性。框架的核心是神经网络近似特征的概念。其主要结果是:所有 nAI 启动功能是普遍性的。事实证明,大多数现有的激活功能都是nAI,因此在Clatia连续功能的空间中是普遍性的。框架具有以下主要特性:首先,它具有从功能分析、概率理论和数字分析中获得的基本手段的建设性框架。其次,这是对大多数现有激活活动都有效的第一个统一框架。第三,作为一个产品,框架为包括Mish、Silus、ELU、GELU、GELU等一些现有激活功能提供了第一个大学的证明。第四,它为我们大多数启动功能提供了新的证明。第五,它以功能分析、概率理论理论和数字分析为基础的神经系统框架提供了新的启动率。第五,它为一个保证了当前虚拟网络框架的普遍性提供了一个新的启动率框架。