Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning

Reinforcement Learning (RL) is a widely-used technique in many domains, including autonomous driving, robotics, stock trading, and video games. Unfortunately, the black box nature of RL agents, combined with increasing legal and ethical considerations, makes it increasingly important that humans understand the reasoning behind the actions taken by an RL agent, particularly in safety-critical domains. To help address this challenge, we introduce PolicyExplainer, a visual analytics interface which lets the user directly query an RL agent. PolicyExplainer visualizes the states, policy, and expected future rewards for an agent, and supports asking and answering questions such as: "Why take this action? Why not this other action? When is this action taken?". PolicyExplainer is designed based upon a domain analysis with RL experts, and is evaluated via empirical assessments on a trio of domains: taxi navigation, an inventory application, and the safety-critical domain of drug recommendation for HIV patients.