Bilevel optimization is a popular two-level hierarchical optimization, which has been widely applied to many machine learning tasks such as hyperparameter learning, meta learning and continual learning. Although many bilevel optimization methods recently have been developed, the bilevel methods are not well studied when the lower-level problem is nonconvex. To fill this gap, in the paper, we study a class of nonconvex bilevel optimization problems, which both upper-level and lower-level problems are nonconvex, and the lower-level problem satisfies Polyak-Lojasiewicz (PL) condition. We propose an efficient momentum-based gradient bilevel method (MGBiO) to solve these deterministic problems. Meanwhile, we propose a class of efficient momentum-based stochastic gradient bilevel methods (MSGBiO and VR-MSGBiO) to solve these stochastic problems. Moreover, we provide a useful convergence analysis framework for our methods. Specifically, under some mild conditions, we prove that our MGBiO method has a sample (or gradient) complexity of $O(\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of the deterministic bilevel problems (i.e., $\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-1})$. Meanwhile, we prove that our MSGBiO and VR-MSGBiO methods have sample complexities of $\tilde{O}(\epsilon^{-4})$ and $\tilde{O}(\epsilon^{-3})$, respectively, in finding an $\epsilon$-stationary solution of the stochastic bilevel problems (i.e., $\mathbb{E}\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-3})$. This manuscript commemorates the mathematician Boris Polyak (1935 -2023).
翻译:双级优化是一种受欢迎的双级优化 { 双级优化 { 双级优化 { 级优化, 已经广泛应用于许多机器学习任务, 如超参数学习、 元学习和持续学习 。 虽然最近开发了许多双级优化方法 。 但是当较低层次的问题不是 convex 时, 双级优化方法没有得到很好的研究 。 为了填补这一空白, 在论文中, 我们研究了一个非civex 双级优化问题的类别, 高层次和低层次的问题都是非 convex, 低层次的问题满足了 Polyak- Lojasiewic( PL) 。 我们提议了一种基于动力的双级( MG-3BO ) 高效的梯级方法来解决这些确定性问题。 我们建议了一种基于动力的双级梯级梯级方法( MSGBi) 和 美元( 美元) 标准( 美元) 。</s>