Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.
翻译:不同的编程技术在社区中广泛使用,是过去几十年机器学习复兴的原因。这些方法虽然强大,但也有局限性。在本简短的报告中,我们讨论了一种基于混乱的常见失败模式,它出现在各种不同的不同环境中,从经常性神经网络和数字物理模拟到培训学习优化者。我们将这一失败追溯到正在研究的系统雅各比人的范围,并为执业者预期这一失败会破坏他们基于差异的优化算法提供标准。