Recent research on accelerated gradient methods of use in optimization has demonstrated that these methods can be derived as discretizations of dynamical systems. This, in turn, has provided a basis for more systematic investigations, especially into the geometric structure of those dynamical systems and their structure--preserving discretizations. In this work, we introduce dynamical systems defined through a contact geometry which are not only naturally suited to the optimization goal but also subsume all previous methods based on geometric dynamical systems. As a consequence, all the deterministic flows used in optimization share an extremely interesting geometric property: they are invariant under contact transformations. In our main result, we exploit this observation to show that the celebrated Bregman Hamiltonian system can always be transformed into an equivalent but separable Hamiltonian by means of a contact transformation. This in turn enables the development of fast and robust discretizations through geometric contact splitting integrators. As an illustration, we propose the Relativistic Bregman algorithm, and show in some paradigmatic examples that it compares favorably with respect to standard optimization algorithms such as classical momentum and Nesterov's accelerated gradient.
翻译:最近关于加速梯度优化使用方法的研究显示,这些方法可以作为动态系统的离散性。这反过来又为更系统的调查提供了基础,特别是这些动态系统的几何结构及其结构保持离散性提供了基础。在这项工作中,我们引入了通过接触几何界定的动态系统,这些系统不仅自然适合优化目标,而且还吸收了基于几何动态系统的所有先前方法。因此,所有优化中使用的确定性流动都具有一个非常有趣的几何属性:它们处于接触变异状态。我们的主要结果是,我们利用这一观察来表明,著名的布雷格曼·汉密尔顿系统能够通过接触变换的方式,变成一个等效但可相互分离的汉密尔顿系统。这反过来又能够通过几何联系分解分解的分解器发展快速和稳健的离性。举例来说,我们建议采用相对相对的布雷格曼算法,并在一些典型的例子中表明,它比得上标准的优化算法,例如古典势头和内斯特洛夫加速的梯度。