Classes of ODE solutions: smoothness, covering numbers, implications for noisy function fitting, and the curse of smoothness phenomenon

Numerical methods for recovering ODE solutions from data largely rely on approximating the solutions using basis functions or kernel functions under a least square criterion. The accuracy of this approach hinges on the smoothness of the solutions. This paper provides a theoretical foundation for these methods by establishing novel results on the smoothness and covering numbers of ODE solution classes (as a measure of their "size"). Our results provide answers to "how do the degree of smoothness and the "size" of a class of ODEs affect the "size" of the associated class of solutions?" We show that: (1) for the first order ODEs, if the absolute values of all $k$th order derivatives are bounded by $1$, then the solution can end up with derivatives whose magnitude grows factorially fast - "a curse of smoothness"; (2) our upper bounds for the covering numbers of the $(\beta+2)-$degree smooth solution classes are greater than those of the "standard" $(\beta+2)-$degree smooth class of univariate functions; (3) the mean squared error of least squares fitting in noisy settings has a convergence rate no larger than $\left(\frac{1}{n}\right)^{\frac{2\left(\beta+2\right)}{2\left(\beta+2\right)+1}}$ if $n=\Omega\left(\left(\beta\sqrt{\log\left(\beta\vee1\right)}\right)^{4\beta+10}\right)$, and under this condition, the rate $\left(\frac{1}{n}\right)^{\frac{2\left(\beta+2\right)}{2\left(\beta+2\right)+1}}$ is minimax optimal in the case of $y^{'}\left(x\right)=f\left(x,\,y\left(x\right)\right)$; (4) more generally, for the higher order Picard type ODEs, $y^{\left(m\right)}\left(x\right)=f\left(x,\,y\left(x\right),\,y^{'}\left(x\right),\,...,y^{\left(m-1\right)}\left(x\right)\right)$, the covering number of the solution class is bounded above by the product of the covering number of the class $\mathcal{F}$ that $f$ ranges over and the covering number of the set where initial values lie.