One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number. Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.
翻译:神经网络的一个令人惊讶的特征是,它们的连接能在多大程度上被切断,而其准确性却几乎没有任何影响。但是当我们跨过一个临界的参数宽度时,任何进一步的缩小都会导致突然的精确性下降。这种下降似乎反映了模型复杂性的丧失,这是我们想要避免的。在这项工作中,我们探索“宽度”会如何影响由神经网络定义的线性区域的几何学,从而减少基于结构的线性区域的预期最大数量。我们观察到,“宽度”也会影响准确性,这与“线性”如何影响线性区域的数量以及我们提议的最大数量一样。相反,我们发现,选择跨层的宽度以最大限度地扩大我们的界限,往往会提高准确性,与所有层的宽度相比较,从而指导我们如何进行“精度”。