Sparse Bayesian Learning (SBL) constructs an extremely sparse probabilistic model with very competitive generalization. However, SBL needs to invert a big covariance matrix with complexity O(M^3 ) (M: feature size) for updating the regularization priors, making it difficult for practical use. There are three issues in SBL: 1) Inverting the covariance matrix may obtain singular solutions in some cases, which hinders SBL from convergence; 2) Poor scalability to problems with high dimensional feature space or large data size; 3) SBL easily suffers from memory overflow for large-scale data. This paper addresses these issues with a newly proposed diagonal Quasi-Newton (DQN) method for SBL called DQN-SBL where the inversion of big covariance matrix is ignored so that the complexity and memory storage are reduced to O(M). The DQN-SBL is thoroughly evaluated on non-linear classifiers and linear feature selection using various benchmark datasets of different sizes. Experimental results verify that DQN-SBL receives competitive generalization with a very sparse model and scales well to large-scale problems.
翻译:SBL 构建了一个极为稀少的概率模型,具有非常竞争性的通用性。然而, SBL 需要将一个具有复杂的 O(MQ3) (M: 特性大小) 的大型共变矩阵倒转,以更新正规化前期,使其难以实际使用。 SBL 有三个问题:(1) 将共变矩阵倒转,在某些情况下可能会获得单一的解决方案,从而妨碍SBL的趋同;(2) 适应高维特征空间或大数据大小问题的能力差;(3) SBL 很容易因大型数据的记忆溢出而受到影响。本文用新提议的SBL 称为 DQN- SBL (DQN) 的双对角 Quasi- Newton (DQN) 方法处理这些问题,该方法忽视了大共变异性矩阵的转换,从而将复杂性和记忆存储缩小到O(M) 。 DQN- SBL 利用不同大小的基准数据集对非线性分类和线性特征选择进行彻底评估。实验结果证实,DN-SBL 问题具有竞争性的一般规模,并具有很高的模型。