Scalable Feature Subset Selection for Big Data using Parallel Hybrid Evolutionary Algorithm based Wrapper in Apache Spark

In this paper, we propose a wrapper for feature subset selection (FSS) based on parallel and distributed hybrid evolutionary algorithms viz., parallel binary differential evolution and threshold accepting (PB-DETA), parallel binary threshold accepting and differential evolution (PB-TADE) under the Apache Spark environment. Here, the FSS is formulated as a combinatorial optimization problem. PB-TADE comprises invoking two optimization algorithms i.e., TA and BDE in tandem in every iteration, while in PB-DETA, BDE is invoked first before TA takes over in tandem in every iteration. In addition to these hybrids, parallel binary differential evolution (P-BDE), is also developed to investigate the role played by TA and for baseline comparison. For all the three proposed approaches, logistic regression (LR) is used to compute the fitness function namely, the area under ROC curve (AUC) score. The effectiveness of the parallel and distributed wrappers is assessed over five large datasets of varying feature space dimension pertaining to the cyber security and biology domains. It is noteworthy that the PB-TADE turned out to be statistically significant compared to P-BDE and PB-DETA. The speed up is reported with respect to the sequential version of the three wrappers. Average AUC score obtained, most repeated feature subsets, feature subsets with least cardinality having best AUC score are also reported. Further, our proposed methods outperformed the state-of-the-art results, wherever the results were reported.