We apply the PAC-Bayes theory to the setting of learning-to-optimize. To the best of our knowledge, we present the first framework to learn optimization algorithms with provable generalization guarantees (PAC-bounds) and explicit trade-off between a high probability of convergence and a high convergence speed. Even in the limit case, where convergence is guaranteed, our learned optimization algorithms provably outperform related algorithms based on a (deterministic) worst-case analysis. Our results rely on PAC-Bayes bounds for general, unbounded loss-functions based on exponential families. By generalizing existing ideas, we reformulate the learning procedure into a one-dimensional minimization problem and study the possibility to find a global minimum, which enables the algorithmic realization of the learning procedure. As a proof-of-concept, we learn hyperparameters of standard optimization algorithms to empirically underline our theory.
翻译:我们把PAC-Bayes理论应用于学习优化的设置。我们最了解的是,我们提出了第一个框架来学习最优化算法,该算法具有可实现的通用保障(PAC-bounds)和高度趋同速度之间的明显权衡。即使在有限的情况下,即使趋同的可能性得到保证,我们所学的优化算法也可以比基于(决定性的)最坏情况分析的与算法相适应。我们的结果依赖于PAC-Bayes的界限,以基于指数式家庭的一般、无限制的损失函数为基础。我们通过概括现有想法,将学习程序重新定位为一维最小化问题,并研究找到一个全球最低要求的可能性,使学习程序的算法化实现。作为概念的证明,我们学习标准优化算法的超参数,以经验方式强调我们的理论。