The deep neural network suffers from many fundamental issues in machine learning. For example, it often gets trapped into a local minimum in training, and its prediction uncertainty is hard to be assessed. To address these issues, we propose the so-called kernel-expanded stochastic neural network (K-StoNet) model, which incorporates support vector regression (SVR) as the first hidden layer and reformulates the neural network as a latent variable model. The former maps the input vector into an infinite dimensional feature space via a radial basis function (RBF) kernel, ensuring absence of local minima on its training loss surface. The latter breaks the high-dimensional nonconvex neural network training problem into a series of low-dimensional convex optimization problems, and enables its prediction uncertainty easily assessed. The K-StoNet can be easily trained using the imputation-regularized optimization (IRO) algorithm. Compared to traditional deep neural networks, K-StoNet possesses a theoretical guarantee to asymptotically converge to the global optimum and enables the prediction uncertainty easily assessed. The performances of the new model in training, prediction and uncertainty quantification are illustrated by simulated and real data examples.
翻译:深心神经网络在机器学习中有许多根本性问题。 例如,它往往被困在培训的当地最低培训点,它的预测不确定性难以评估。为了解决这些问题,我们提议了所谓的内核扩展神经神经网络(K-StoNet)模型,该模型将支持矢量回归(SVR)作为第一个隐藏的层,并将神经网络改制为潜在的变数模型。前一模型将输入矢量通过辐射基函数(RBF)内核将输入到一个无限的天体特征空间中,确保培训损失表面没有当地微型数据。后一模型将高度的非convex神经网络培训问题打破为一系列低度convex优化问题,便于评估其预测不确定性。K-StoNet可以很容易地使用浸入式-正规化优化(IRO)算法来培训。与传统的深线网络相比,K-StoNet拥有理论保证,以便与全球最佳化的模型相近似一致,并使预测真实的不确定性能够很容易地评估。通过模拟的模型和模拟数据模拟的性模型的性表现。