AUC: Nonparametric Estimators and Their Smoothness

Nonparametric estimation of a statistic, in general, and of the error rate of a classification rule, in particular, from just one available dataset through resampling is well mathematically founded in the literature using several versions of bootstrap and influence function. This article first provides a concise review of this literature to establish the theoretical framework that we use to construct, in a single coherent framework, nonparametric estimators of the AUC (a two-sample statistic) other than the error rate (a one-sample statistic). In addition, the smoothness of some of these estimators is well investigated and explained. Our experiments show that the behavior of the designed AUC estimators confirms the findings of the literature for the behavior of error rate estimators in many aspects including: the weak correlation between the bootstrap-based estimators and the true conditional AUC; and the comparable accuracy of the different versions of the bootstrap estimators in terms of the RMS with little superiority of the .632+ bootstrap estimator.