In this paper, we carry out numerical analysis to prove convergence of a novel sample-wise back-propagation method for training a class of stochastic neural networks (SNNs). The structure of the SNN is formulated as discretization of a stochastic differential equation (SDE). A stochastic optimal control framework is introduced to model the training procedure, and a sample-wise approximation scheme for the adjoint backward SDE is applied to improve the efficiency of the stochastic optimal control solver, which is equivalent to the back-propagation for training the SNN. The convergence analysis is derived with and without convexity assumption for optimization of the SNN parameters. Especially, our analysis indicates that the number of SNN training steps should be proportional to the square of the number of layers in the convex optimization case. Numerical experiments are carried out to validate the analysis results, and the performance of the sample-wise back-propagation method for training SNNs is examined by benchmark machine learning examples.
翻译:在本文中,我们进行了数字分析,以证明用于培训一批神经网络的新型样样反向分析方法(SNN)的趋同。SNN的结构是作为随机差异方程式(SDE)的分解而拟订的。引入了一种随机最佳控制框架来模拟培训程序,并应用了联合后向SDE的样比近似方法来提高随机最佳控制求解器的效率,该方法相当于培训SNN的后向分析。聚合分析与SNN参数优化的混合假设同时得出,而不是一致假设。特别是,我们的分析表明,SNNN培训步骤的数量应该与Convex优化案例中层数的正方形成比例成比例。进行了数值实验,以验证分析结果,并且通过基准机器学习实例对培训SNN的样比法反向正确方法的性能进行了审查。