In the paper, we propose a class of faster adaptive gradient descent ascent methods for solving the nonconvex-strongly-concave minimax problems by using unified adaptive matrices used in the SUPER-ADAM \citep{huang2021super}. Specifically, we propose a fast adaptive gradient decent ascent (AdaGDA) method based on the basic momentum technique, which reaches a low sample complexity of $O(\kappa^4\epsilon^{-4})$ for finding an $\epsilon$-stationary point without large batches, which improves the existing result of adaptive minimax optimization method by a factor of $O(\sqrt{\kappa})$. Moreover, we present an accelerated version of AdaGDA (VR-AdaGDA) method based on the momentum-based variance reduced technique, which achieves the best known sample complexity of $O(\kappa^3\epsilon^{-3})$ for finding an $\epsilon$-stationary point without large batches. Further assume the bounded Lipschitz parameter of objective function, we prove that our VR-AdaGDA method reaches a lower sample complexity of $O(\kappa^{2.5}\epsilon^{-3})$ with the mini-batch size $O(\kappa)$. In particular, we provide an effective convergence analysis framework for our adaptive methods based on unified adaptive matrices, which include almost existing adaptive learning rates.
翻译:在本文中,我们建议了一种更快速的适应性梯度下游方法,通过使用SUPER-ADAM \ citep{huang2021Suer}中使用的统一适应矩阵来解决非稳定型小型马克思问题。具体地说,我们建议了一种基于基本动力技术的快速适应性梯度体面上升方法(AdaGDA),该方法的样本复杂性低,达到美元(kappa_4\epsilon}-4}美元(美元)的已知最佳样本复杂性,用于在没有大批量的情况下寻找一个美元(epsilon)的固定点,从而通过一个美元(sqrt_kapta})的系数来改善适应性小型小马克思优化方法的现有结果。此外,我们根据动力差异降低技术,提出了快速适应性梯度(VR-ADGDA)方法的加速版本,该方法达到了已知的美元(k) 美元( weppa_3\epsilon ⁇ -3} (美元) 用于找到一个没有大批量的固定点的美元固定点,这可以改善微压优化优化优化方法的现有结果。 进一步假设ALIA- dislev=xxxxxxxxxxxxxx