The understanding about the minimum width of deep neural networks needed to ensure universal approximation for different activation functions has progressively been extended (Park et al., 2020). In particular, with respect to approximation on general compact sets in the input space, a network width less than or equal to the input dimension excludes universal approximation. In this work, we focus on network functions of width less than or equal to the latter critical bound. We prove that in this regime, the exact fit of partially constant functions on disjoint compact sets is still possible for ReLU network functions under some conditions on the mutual location of these components. We show that with cosine as activation function, a three layer network of width one is sufficient to approximate any function on arbitrary finite sets. Conversely, we prove a maximum principle from which we conclude that for all continuous and monotonic activation functions, universal approximation of arbitrary continuous functions is impossible on sets that coincide with the boundary of an open set plus an inner point.
翻译:对确保不同激活功能的通用近似所需的深神经网络最小宽度的理解已逐步扩大(Park等人,2020年);特别是,关于输入空间一般紧凑装置的近似度,一个小于或等于输入维度的网络宽度排除了通用近似值。在这项工作中,我们侧重于宽度小于或等于后一关键约束的网络功能。我们证明,在这个制度中,在不连接的紧凑装置上部分恒定功能的准确性仍有可能在这些部件的相互位置的某些条件下用于RELU网络功能。我们显示,以连线作为激活功能,一个三层宽的网络足以匹配任意定点的任何功能。相反,我们证明了一个最大原则,我们据此得出结论,对于所有连续和单调的激活功能,在与开放装置的边界加上内部点一致的设置上,无法对任意连续功能的普遍接近。