Unrolled computation graphs arise in many scenarios, including training RNNs, tuning hyperparameters through unrolled optimization, and training learned optimizers. Current approaches to optimizing parameters in such computation graphs suffer from high variance gradients, bias, slow updates, or large memory usage. We introduce a method called Persistent Evolution Strategies (PES), which divides the computation graph into a series of truncated unrolls, and performs an evolution strategies-based update step after each unroll. PES eliminates bias from these truncations by accumulating correction terms over the entire sequence of unrolls. PES allows for rapid parameter updates, has low memory usage, is unbiased, and has reasonable variance characteristics. We experimentally demonstrate the advantages of PES compared to several other methods for gradient estimation on synthetic tasks, and show its applicability to training learned optimizers and tuning hyperparameters.
翻译:在许多设想中,包括培训RNN,通过不滚动优化调整超参数,以及培训学习到的优化。目前优化此类计算图参数的方法存在高差异梯度、偏差、缓慢更新或大量记忆用量。我们采用了一种称为“持续进化战略”的方法,将计算图分为一系列短线脱轨的脱轨,并在每次脱轨后执行一个基于进化战略的更新步骤。PES通过将校正术语积聚于整个非滚动序列中,消除了这些脱轨中的偏差。PES允许快速更新参数,记忆用量低,不偏向,并具有合理的差异特性。我们实验性地展示了PES相对于合成任务其他若干梯度估算方法的优势,并展示了其对培训学习优化和调整超参数的实用性。