Accelerated Gradient Methods for Sparse Statistical Learning with Nonconvex Penalties

Nesterov's accelerated gradient (AG) is a popular technique to optimize objective functions comprising two components: a convex loss and a penalty function. While AG methods perform well for convex penalties, such as the LASSO, convergence issues may arise when it is applied to nonconvex penalties, such as SCAD. A recent proposal generalizes Nesterov's AG method to the nonconvex setting but has never been applied to sparse statistical learning problems. There are several hyperparameters to be set before running the proposed algorithm. However, there is no explicit rule as to how the hyperparameters should be selected. In this article, we consider the application of this nonconvex AG algorithm to high-dimensional linear and logistic sparse learning problems, and propose a hyperparameter setting based on the complexity upper bound to accelerate convergence. We further establish the rate of convergence and present a simple and useful bound for the damping sequence. Simulation studies show that convergence can be made, on average, considerably faster than that of the conventional ISTA algorithm. Our experiments also show that the proposed method generally outperforms the current state-of-the-art method in terms of signal recovery.