Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers \emph{reverse-mode} AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of \emph{linear factoring} to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. We demonstrate the practical use of our technique by providing a performant implementation that differentiates most of Haskell98.
翻译:当双数前式自动差异化(AD)配对时,如果双数正切值的双数 \ emph{ verse- mode} AD 试图使用一个类似的简单想法实现反向反向反向反向: 将每个斜值配对为反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向反向