Observations on Recurrent Loss in the Neural Network Model of a Partial Differential Equation: the Advection-Diffusion Equation

A growing body of literature has been leveraging techniques of machine learning (ML) to build novel approaches to approximating the solutions to partial differential equations. Noticeably absent from the literature is a systematic exploration of the stability of the solutions generated by these ML approaches. Here, a recurrent network is introduced that matches precisely the evaluation of a multistep method paired with a collocation method for approximating spatial derivatives in the advection diffusion equation. This allows for two things: 1) the use of traditional tools for analyzing the stability of a numerical method for solving PDEs and 2) bringing to bear efficient techniques of ML for the training of approximations for the action of (spatial) linear operators. Observations on impacts of varying the large number of parameters in even this simple linear problem are presented. Further, it is demonstrated that stable solutions can be found even where traditional numerical methods may fail.