Sparse neural networks have received increasing interest due to their small size compared to dense networks. Nevertheless, most existing works on neural network theory have focused on dense neural networks, and the understanding of sparse networks is very limited. In this paper, we study the loss landscape of one-hidden-layer sparse networks. First, we consider sparse networks with a dense final layer. We show that linear networks can have no spurious valleys under special sparse structures, and non-linear networks could also admit no spurious valleys under a wide final layer. Second, we discover that spurious valleys and spurious minima can exist for wide sparse networks with a sparse final layer. This is different from wide dense networks which do not have spurious valleys under mild assumptions.
翻译:与稠密的网络相比,松散的神经网络由于规模小而受到越来越多的关注。然而,大部分现有的神经网络理论工程都集中在密集的神经网络上,对稀疏网络的了解非常有限。在本论文中,我们研究了一层稀疏网络的消失情况。首先,我们考虑的是具有稠密最后层的稀疏网络。我们表明,在特殊的稀疏结构下,线性网络不可能有虚假的山谷,非线性网络也可能在宽广的最后一层下没有虚假的山谷。第二,我们发现虚假的峡谷和虚假的迷你马可以存在于具有稀疏最后层的广稀疏网络上。这不同于在温和假设下没有虚假山谷的大密度网络。