Risk-Sensitive Reinforcement Learning

The classic objective in a reinforcement learning (RL) problem is to find a policy that minimizes, in expectation, a long-run objective such as the infinite-horizon cumulative discounted or long-run average cost. In many practical applications, optimizing the expected value alone is not sufficient, and it may be necessary to include a risk measure in the optimization process, either in the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., variance, exponential utility, percentile performance, chance constraints, value at risk (quantile), conditional value-at-risk, coherent risk measure, prospect theory and its later enhancement, cumulative prospect theory. In this article, we focus on the combination of risk criteria and reinforcement learning in a constrained optimization framework, i.e., a setting where the goal to find a policy that optimizes the usual objective of infinite-horizon discounted/average cost, while ensuring that an explicit risk constraint is satisfied. We introduce the risk-constrained RL framework, cover popular risk measures based on variance, conditional value-at-risk, and chance constraints, and present a template for a risk-sensitive RL algorithm. Next, we study risk-sensitive RL with the objective of minimizing risk in an unconstrained framework, and cover cumulative prospect theory and coherent risk measures as special cases. We survey some of the recent work on this topic, covering problems encompassing discounted cost, average cost, and stochastic shortest path settings, together with the aforementioned risk measures, in constrained as well as unconstrained frameworks. This non-exhaustive survey is aimed at giving a flavor of the challenges involved in solving risk-sensitive RL problems, and outlining some potential future research directions.