Estimation of Over-parameterized Models from an Auto-Modeling Perspective

From a model-building perspective, in this paper we propose a paradigm shift for fitting over-parameterized models. Philosophically, the mindset is to fit models to future observations rather than the observed sample. Technically, choosing an imputation model for generating future observations, we fit over-parameterized models to future observations via optimizing an approximation to the desired expected loss function based on its sample counterpart and what we call an adaptive \it {dual function}. This technique is discussed in detail for both creating bootstrap imputation and final estimation with bootstrap imputation. The method is illustrated with the many-normal-means problem, $n < p$ linear regression, and deep convolutional neural networks for image classification of MNIST digits. The numerical results demonstrate superior performance across these three different types of applications. For example, For the many-normal-means problem, our method uniformly dominates James-Stein and Efron's $g-$modeling, and for the MNIST image classification, it shows state-of-the-art performance in common model structures. While this paper is largely expository because of the ambitious task of taking a look at over-parameterized models from a new perspective, fundamental theoretical properties are also investigated. We conclude the paper with a few remarks.