Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks. However, the direct mapping from states to actions makes it hard to interpret the rationale behind the decision-making of the agents. In contrast to previous a-posteriori methods for visualizing DeepRL policies, in this work, we propose to equip the DeepRL model with an innate visualization ability. Our proposed agent, named region-sensitive Rainbow (RS-Rainbow), is an end-to-end trainable network based on the original Rainbow, a powerful deep Q-network agent. It learns important regions in the input domain via an attention module. At inference time, after each forward pass, we can visualize regions that are most important to decision-making by backpropagating gradients from the attention module to the input frames. The incorporation of our proposed module not only improves model interpretability, but leads to performance improvement. Extensive experiments on games from the Atari 2600 suite demonstrate the effectiveness of RS-Rainbow.
翻译:深度强化学习(DeepRL)代理在许多任务中超越了人类水平的表现。然而,从状态到行动的直接映射使得解释代理的决策过程变得困难。与以前的事后方法不同,用于可视化DeepRL策略,在本文中,我们提出了一种具有先天可视化能力的DeepRL模型。我们提出的代理名为区域敏感Rainbow(RS-Rainbow),是一种基于原始Rainbow,一种强大的深度Q网络代理的端到端可训练网络。它通过一个注意模块学习输入域中的重要区域。在推理时间,每个前向时间通过从注意模块到输入帧反向传播梯度,我们可以将最重要的决策制定区域可视化。我们的提议模块的纳入不仅提高了模型的可解释性,还引导了性能的提高。对Atari 2600游戏套件的广泛实验证明了RS-Rainbow的有效性。