On the Search for Feedback in Reinforcement Learning

This paper addresses the problem of learning the optimal feedback policy for a nonlinear stochastic dynamical system. Feedback policies typically need a high dimensional parametrization, which makes Reinforcement Learning (RL) algorithms that search for an optimum in this large parameter space, sample inefficient and subject to high variance. We propose a "decoupling" principle that drastically reduces the feedback parameter space while still remaining locally optimal. A corollary of this result is a decoupled data-based control (D2C) algorithm for RL: first, an open-loop deterministic trajectory optimization problem is solved using a black-box simulation model of the dynamical system. Then, a linear closed-loop control is developed around this nominal trajectory using the simulation model. Empirical evidence suggests highly significant reduction in training time, as well as the training variance, without compromising on performance, compared to state of the art RL algorithms.