Analyzing and Quantifying Generalization in Convolutional Neural Networks

Generalization is the key capability of convolutional neural networks (CNNs). However, it is still quite elusive for differentiating the CNNs with good or poor generalization. It results in the barrier for providing reliable quantitative measure of generalization ability. To this end, this paper aims to clarify the generalization status of individual units in typical CNNs and quantify the generalization ability of networks using image classification task with multiple classes data. Firstly, we propose a feature quantity, role share, consisting of four discriminate statuses for a certain unit based on its contribution to generalization. The distribution of role shares across all units provides a straightforward visualization for the generalization of a network. Secondly, using only training sets, we propose a novel metric for quantifying the intrinsic generalization ability of networks. Lastly, a predictor of testing accuracy via only training accuracy of typical CNN is given. Empirical experiments using practical network model (VGG) and dataset (ImageNet) illustrate the rationality and effectiveness of our feature quantity, metric and predictor.