PLSS: A Projected Linear Systems Solver

We propose iterative projection methods for solving square or rectangular consistent linear systems $Ax = b$. Projection methods use sketching matrices (possibly randomized) to generate a sequence of small projected subproblems, but even the smaller systems can be costly. We develop a process that appends one column each iteration to the sketching matrix and that converges in a finite number of iterations independent of whether the sketch is random or deterministic. In general, our process generates orthogonal updates to the approximate solution $x_k$. By choosing the sketch to be the set of all previous residuals, we obtain a simple recursive update and convergence in at most rank($A$) iterations (in exact arithmetic). By choosing a sequence of identity columns for the sketch, we develop a generalization of the Kaczmarz method. In experiments on large sparse systems, our method (PLSS) with residual sketches is competitive with LSQR, and our method with residual and identity sketches compares favorably to state-of-the-art randomized methods.