Tannenbaum's gain-margin optimization meets Polyak's heavy-ball algorithm

The paper highlights a relatively unknown link between algorithm design in optimization and control synthesis in robust control. Specifically, quadratic optimization can be recast as a regulation problem within the framework of $\mathcal{H}_\infty$ control. From this vantage point, the optimality of Polyak's fastest heavy-ball algorithm can be ascertained as a solution to a gain margin optimization problem. The approach is independent of Polyak's original and brilliant argument, yet simpler, and relies on the foundational work by Tannenbaum that introduced and solved the gain margin optimization via Nevanlinna--Pick interpolation theory. The link between first-order optimization methods and robust control theory sheds new light into limits of algorithmic performance for such methods, and suggests a new framework where similar computational problems can be systematically studied and algorithms optimized. In particular, it raises the question as to whether periodically scheduled algorithms can achieve faster rates for quadratic optimization, in a manner analogous to periodic control that extends gain margin beyond that of time-invariant control. This turns out not to be the case, due to the analytic obstruction of a transmission zero that is inherent in causal optimization algorithms. Interestingly, this obstruction can be removed with implicit algorithms, cast in a similar manner as feedback regulation problems with causal, but not strictly causal dynamics, thereby devoid of the transmission zero at infinity and able to achieve superior convergence rates. The confluence of the fields of optimization algorithms and control provides a frame to tackle questions pertaining to speed, accuracy, distributed computation, and so forth, and to delineate respective limits to performance and tradeoffs in a systematic manner, utilizing the formalism of robust control.