Model Averaging for Support Vector Machine by J-fold Cross-Validation

Support vector machine (SVM) is a classical tool to deal with classification problems, which is widely used in biology, statistics and machine learning and good at small sample size and high-dimensional situation. This paper proposes a model averaging method, called SVMMA, to address the uncertainty from deciding which covariates should be included for SVM and to promote its prediction ability. We offer a criterion to search the weights to combine many candidate models that are composed of different parts from the total covariates. To build up the candidate model set, we suggest to use a screening-averaging form in practice. Especially, the model averaging estimator is proved to be asymptotically optimal in the sense of achieving the lowest hinge risk among all possible combination. Finally, we do some simulation to compare the proposed model averaging method with several other model selection/averaging and ensemble learning methods, and apply to four real datasets.