The Recurrent Neural Tangent Kernel

The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs, including how different time steps are weighted by the RNTK to form the output under different initialization parameters and nonlinearity choices, and how inputs of different lengths are treated. The ability to compare inputs of different length is a property of RNTK that should greatly benefit practitioners. We demonstrate via a synthetic and 56 real-world data experiments that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.