Dynamic Sparse Training for Deep Reinforcement Learning

Dynamic sparse training (DST) literature demonstrates that a highly sparse neural network can match the performance of its corresponding dense network in supervised and unsupervised learning when it is trained from scratch while substantially reducing the computational and memory costs. In this paper, we show for the first time that deep reinforcement learning can also benefit from dynamic sparse training. We demonstrate that DST can be leveraged to decrease the long training time required by deep reinforcement learning agents without sacrificing performance. To achieve this, we propose a DST algorithm that adapts to the online nature and instability of the deep reinforcement learning paradigm. We integrate our proposed algorithm with state-of-the-art deep reinforcement learning methods. Experimental results demonstrate that our dynamic sparse compact agents can effectively learn and achieve higher performance than the original dense methods while reducing the parameter count and floating-point operations (FLOPs) by 50%. More impressively, our dynamic sparse agents have a faster learning speed. They can reach the final performance achieved by dense agents after 40-50% of the steps required by the latter. We evaluate our approach on OpenAI gym continuous control tasks.