Interpreting Distributional Reinforcement Learning: Regularization and Optimization Perspectives

Distributional reinforcement learning~(RL) is a class of state-of-the-art algorithms that estimate the whole distribution of the total return rather than only its expectation. Despite the remarkable performance of distributional RL, a theoretical understanding of its advantages over expectation-based RL remains elusive. In this paper, we illuminate the superiority of distributional RL from both regularization and optimization perspectives. Firstly, by applying an expectation decomposition, the additional impact of distributional RL compared with expectation-based RL is interpreted as a \textit{risk-aware entropy regularization} in the \textit{neural Z-fitted iteration} framework. We also provide a rigorous comparison between the resulting entropy regularization and the vanilla one in maximum entropy RL. Through the lens of optimization, we shed light on the stability-promoting distributional loss with desirable smoothness properties in distributional RL. Moreover, the acceleration effect of distributional RL owing to the risk-aware entropy regularization is also provided. Finally, rigorous experiments reveal the different regularization effects as well as the mutual impact of vanilla entropy and risk-aware entropy regularization in distributional RL, focusing specifically on actor-critic algorithms. We also empirically verify that the distributional RL algorithm enjoys a more stable gradient behavior, contributing to its stable optimization and acceleration effect as opposed to classical RL. Our research paves a way towards better interpreting the superiority of distributional RL algorithms.