Greedy Training Algorithms for Neural Networks and Applications to PDEs

Recently, neural networks have been widely applied for solving partial differential equations (PDEs). However, with current training algorithms the numerical convergence of neural networks when solving PDEs has not been empirically observed. The primary difficulty lies in solving the highly non-convex optimization problems resulting from the neural network discretization. Theoretically analyzing the optimization process presents significant difficulties and empirical experiments require extensive hyperparameter tuning to achieve acceptable results. In order to conquer this challenge, we develop a novel greedy training algorithm for shallow neural networks in this paper. We also analyze the resulting method and obtain a priori error bounds when solving PDEs from the function class defined by shallow networks. This rigorously establishes the convergence of the method as the network size increases. Finally, we test the algorithm on several benchmark examples, including high dimensional PDEs, to confirm the theoretical convergence rate and to establish its efficiency and robustness. An advantage of this method is its straightforward applicability to high-order equations on general domains.