Warming-up recurrent neural networks to maximize reachable multi-stability greatly improves learning

Training recurrent neural networks is known to be difficult when time dependencies become long. Consequently, training standard gated cells such as the gated recurrent unit (GRU) and the long short-term memory (LSTM) on benchmarks where long-term memory is required remains an arduous task. In this work, we show that although most classical networks have only one stable equilibrium at initialisation, learning on tasks that require long-term memory only occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long-term dependencies difficult. This insight leads to the design of a novel, general way to initialise any recurrent network connectivity through a procedure called "warmup" to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of attractors within the network that can be reached through relevant input trajectories. Warming up is performed before training, using stochastic gradient descent on a specifically designed loss. We show on information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves recurrent neural network performance for multiple recurrent cell types, but sometimes impedes precision. We therefore introduce a parallel recurrent network structure with a partial warmup that is shown to greatly improve learning of long-term dependencies in sequences while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell type when long-term memory is required.