This work explores the Benevolent Training Hypothesis (BTH) which argues that the complexity of the function a deep neural network (NN) is learning can be deduced by its training dynamics. Our analysis provides evidence for BTH by relating the NN's Lipschitz constant at different regions of the input space with the behavior of the stochastic training procedure. We first observe that the Lipschitz constant close to the training data affects various aspects of the parameter trajectory, with more complex networks having a longer trajectory, bigger variance, and often veering further from their initialization. We then show that NNs whose 1st layer bias is trained more steadily (i.e., slowly and with little variation) have bounded complexity even in regions of the input space that are far from any training point. Finally, we find that steady training with Dropout implies a training- and data-dependent generalization bound that grows poly-logarithmically with the number of parameters. Overall, our results support the intuition that good training behavior can be a useful bias towards good generalization.
翻译:这项工作探索了福利培训假说(BTH),认为深神经网络(NN)正在学习的功能的复杂性可以通过培训动态推断出来。我们的分析通过将输入空间不同区域的NN的Lipschitz常数与随机培训程序的行为联系起来,为BTH提供了证据。我们首先发现,Lipschitz的常数接近培训数据会影响参数轨迹的各个方面,而更复杂的网络的轨道较长,差异更大,而且往往从初始化开始就更进一步。我们随后表明,即使在远离任何培训点的输入空间的区域,其一层偏差得到更稳定的训练(即缓慢和几乎没有变异)的非NNCs的复杂程度也相交织。最后,我们发现,与辍学有关的稳步培训意味着一个培训和数据依赖的概括性约束,与参数数成多对数。总体而言,我们的结果支持了这样的直觉,即良好的培训行为可能是向好的一般化方向的有用偏差。