前馈神经网络的结构稀疏化设计与分析

项目名称： 前馈神经网络的结构稀疏化设计与分析

项目编号： No.61473059

项目类型： 面上项目

立项/批准年度： 2015

项目学科： 其他

项目作者： 吴微

作者单位： 大连理工大学

项目金额： 80万元

中文摘要： 在保证适当学习精度前提下，神经网络的权值连接以及神经元应该尽可能少（结构稀疏化），从而降低成本，提高稳健性和推广精度。本项目用正则化方法研究前馈神经网络的结构稀疏化，有以下几个要点：1）传统的神经网络正则化通过惩罚冗余权值连接达到权值稀疏化；我们主张通过惩罚冗余单元而达到效率更高的单元稀疏化。2）除了传统的用于稀疏化的L1正则化之外，我们还采用近几年流行的L1/2正则化。为了解决L1/2正则化算子不光滑，容易导致迭代过程震荡这一问题，我们试图在不光滑点的一个小邻域内采用磨光技巧，构造一种光滑化L1/2正则化算子，以期达到比L1正则化更高的稀疏化效率。3）我们首倡研究输入层单元稀疏化，不但作为整个网络结构稀疏化的一部分，更使得神经网络成为非线性压缩感知的一个可行工具。4）用于多分类问题时，我们首倡输出层单元采用二进制方式，代替传统的亮灯方式，简单高效地减少输出单元。

中文关键词： 神经网络；稀疏化；正则化

英文摘要： On the premise of appropriate learning accuracy, the number of the neurons and weights of a neural network should be as less as possible (constructional sparsification), so as to reduce the cost, and to improve the robustness and the generalization accuracy. This project studies the constructional sparcification of feedforward neural networks by using regularization methods, and it contains the following main points: 1) We propose to punish the redundant neurons to get a more effective nodes sparsification, while the traditional approach punishes the weights to get weight sparcification. 2) Apart from the traditional L1 regularization for sparsification, we also use L1/2 regularization. To remove the oscillation in the iteration process due to the nonsmoothness of the L1/2 regularizer, we propose to smooth it in a neighborhood of the nonsmooth point to get a smoothing L1/2 regularizer. By doing so, we expect to improve the efficiency of the L1/2 regularizer so as to surpass the L1 regularizer. 2) We propose to study the sparsification of the input layer neurons. It not only is a part of the whole network sparsification, but also provides a tool for the sparsification of variables of general nonlinear problems. 3) For the output representation for multi-classification problems, we propose to use the binary approach to replace the traditional one-for-each approach so as to simply and effectively reduce the number of the output neurons.

英文关键词： Neural networks;sparsification;regularization