Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.
翻译:在结构化参数空间(例如低级共变)上的自然渐变(NGD)在计算上由于难以计算而具有挑战性。我们通过使用 emph{当地参数坐标 来解决这个问题。我们通过使用 emph{当地参数坐标 来获得灵活而有效的NGD 方法,该方法对于结构化参数化的广泛特性非常有效。我们展示了四个应用方法:(1) 概括指数性自然进化战略,(2) 恢复现有的牛顿式算法,(3) 产生新的结构化第二级算法,(4) 提供新的算法,学习高山和Wishart分布法的共变法。我们展示了从深层次学习、变异推论和演进战略等一系列问题的结果。我们的工作为可扩展结构化的几何方法开辟了新的方向。