Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks. In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs are able to fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.
翻译:神经切核内核( NTK) 是分析神经网络及其一般界限的培训动态的有力工具。 关于 NTK 的研究致力于典型的神经网络结构,但对于使用Hadamard 产品的神经网络(NNS-Hp),例如StylesGAN 和多光谱神经网络的神经网络来说是不完整的。 在这项工作中,我们为NNS-Hp 的特殊类别,即多核神经网络,得出了NTK 的有限和宽度的NTK 配方。 我们证明它们与相关的NTK 内核回归预测器是等同的,NTK 扩大了应用范围。 根据我们的成果,我们澄清了PNNNP在标准神经网络上的分离,与标准神经网络和光谱偏差有关。我们的两个关键见解是,与标准的神经网络相比,PNNC能够将更复杂的功能纳入外控系统,并承认各自NTK 的精度衰减。 此外,我们的理论结果可以扩大到更深的NTH, 将我们的NT-H 的实验结构扩展到更深层次,从而提供我们对NTS-H 进行更深入的验证。