On Residual Minimization for PDEs: Failure of PINN, Modified Equation, and Implicit Bias

As a popular and easy-to-implement machine learning method for solving differential equations, the physics-informed neural network (PINN) sometimes may fail and find poor solutions which bias against the exact ones. In this paper, we establish a framework of modified equation to explain the failure phenomenon and characterize the implicit bias of a general residual minimization (RM) method. We provide a simple way to derive the modified equation which models the numerical solution obtained by RM methods. Next, we show the modified solution deviates from the original exact solution. The proof uses a by-product of this paper, that is, a necessary and sufficient condition on characterizing the singularity of the coefficients. This equivalent condition can be extended to other types of equations in the future. Finally, we prove, as a complete characterization of the implicit bias, that RM method implicitly biases the numerical solution against the exact solution and towards a modified solution. In this work, we focus on elliptic equations with discontinuous coefficients, but our approach can be extended to other types of equations and our understanding of the implicit bias may shed light on further development of deep learning based methods for solving equations.