Robust Prediction Error Estimation with Monte-Carlo Methodology

In predictive modeling with simulation or machine learning, it is critical to assess the quality of estimated values through output analysis accurately. In recent decades output analysis has become enriched with methods that quantify the impact of input data uncertainty in the model outputs to increase robustness. However, most developments apply when the input data can be parametrically parameterized. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling. This framework provides nonparametric quantification of the variance and bias induced in the outputs with higher-order accuracy. Our new bias-corrected estimation from the model outputs leverages the extension of fast iterative bootstrap sampling and higher-order influence functions. For the scalability of the proposed estimation methods, we devise budget-optimal rules and leverage control variates for variance reduction. Our numerical results demonstrate a clear advantage in building better and more robust confidence intervals for both simulation and machine learning frameworks.