A deluge of recent work has explored equivalences between wide neural networks and kernel methods. A central theme is that one can analytically find the kernel corresponding to a given wide network architecture, but despite major implications for architecture design, no work to date has asked the converse question: given a kernel, can one find a network that realizes it? We affirmatively answer this question for fully-connected architectures, completely characterizing the space of achievable kernels. Furthermore, we give a surprising constructive proof that any kernel of any wide, deep, fully-connected net can also be achieved with a network with just one hidden layer and a specially-designed pointwise activation function. We experimentally verify our construction and demonstrate that, by just choosing the activation function, we can design a wide shallow network that mimics the generalization performance of any wide, deep, fully-connected network.
翻译:最近一连串的工作探索了广泛的神经网络和内核方法之间的等同性。一个中心主题是,人们可以分析找到与特定宽网络结构相对应的内核,但尽管对建筑设计有重大影响,迄今为止没有工作提出相反的问题:鉴于内核,我们能否找到一个认识到它的网络?我们肯定地回答这个问题,以完全连接的建筑为主,完全地说明可实现内核的空间。此外,我们提供了令人惊讶的建设性证据,证明任何宽、深、完全连接的网络的内核,只要有一个隐藏层的网络和专门设计的点感应功能,就可以实现。我们实验性地核查我们的构造,并证明只要选择激活功能,我们就能设计一个宽浅的网络,模仿任何宽广、深、完全连接的网络的通用性能。