Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove that for two-layer fully connected neural networks with large hidden nodes, randomly initialized ISGD converges to a globally optimal solution for the quadratic loss function. Empirical results demonstrate that ISGD works well in practice and compares favorably to other gradient-based optimization methods such as SGD and Adam, while can also effectively address the numerical stiffness in training dynamics via gradient descent.
翻译:物理知情神经网络(PINNs)在解决前方和反面差异方程式问题方面得到了有效证明,但是,当目标功能近似于高频或多尺度特征时,它们仍陷于培训失败之中。在本文件中,我们提议采用隐含的随机梯度梯度下降(ISGD)方法来培训PINNs,以提高培训过程的稳定性。我们用粗略分析ISGD如何克服PIN的梯度流动动态的僵硬性,特别是多尺度解决方案的问题。我们理论上证明,对于两层完全连接的神经网络来说,与大型隐藏节点、随机初始化的ISGD相连接的神经网络,会汇集到一种全球最佳的四方损失功能解决方案。经验性结果显示,ISGD在实践上行之有效,并与其他梯度优化方法如SGD和Adam相比是有利的,同时能够有效地解决通过梯度下降在培训动态中的数字僵硬性。</s>