This paper focuses on stochastic methods for solving smooth non-convex strongly-concave min-max problems, which have received increasing attention due to their potential applications in deep learning (e.g., deep AUC maximization, distributionally robust optimization). However, most of the existing algorithms are slow in practice, and their analysis revolves around the convergence to a nearly stationary point. We consider leveraging the Polyak-\L ojasiewicz (PL) condition to design faster stochastic algorithms with stronger convergence guarantee. Although PL condition has been utilized for designing many stochastic minimization algorithms, their applications for non-convex min-max optimization remain rare. In this paper, we propose and analyze a generic framework of proximal epoch-based method with many well-known stochastic updates embeddable. Fast convergence is established in terms of both {\bf the primal objective gap and the duality gap}. Compared with existing studies, (i) our analysis is based on a novel Lyapunov function consisting of the primal objective gap and the duality gap of a regularized function, and (ii) the results are more comprehensive with improved rates that have better dependence on the condition number under different assumptions. We also conduct deep and non-deep learning experiments to verify the effectiveness of our methods.
翻译:本文侧重于解决平滑的非混凝土强凝固的微轴问题的随机方法,这些问题因其在深层学习中的潜在应用而日益受到重视(如深层AUC最大化、分布强力优化等),然而,大多数现有算法在实践中进展缓慢,其分析围绕接近固定点的趋同点。我们考虑利用Polyak-Lojasiewicz(PL)条件来设计更快的随机算法,并有更强的趋同保证。尽管PL条件被用于设计许多随机最小化算法,但非对混凝土微轴优化的应用仍然很少。在本文件中,我们提出和分析一个基于准亚氏法方法的通用框架,许多众所周知的随机化更新可以嵌入其中。我们从原始目标差距和双重性差距的角度建立了快速的趋同点。与现有的研究相比,(i)我们的分析基于由原始目标差距和二元性优化法组成的新型Lyapunov功能,而原始目标差距和二元性优化的双重性假设在常规性实验中也存在更好的可靠性差距。