Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

The elastic net combines lasso and ridge regression to fuse the sparsity property of lasso with the grouping property of ridge regression. The connections between ridge regression and gradient descent and between lasso and forward stagewise regression have previously been shown. Similarity to how the elastic net generalizes lasso and ridge regression, we introduce elastic gradient descent, a generalization of gradient descent and forward stagewise regression. We compare elastic gradient descent to the elastic net on real and simulated data, and show that it provides similar solution paths, but is of several orders of magnitude faster. Compared to forward stagewise regression, elastic gradient descent selects a model that, although still sparse, provides considerable lower prediction and estimation errors. We also investigate the case of infinitesimal step size, obtaining a piecewise analytical solution we refer to as elastic gradient flow.