Coagent Networks Revisited

This work is aiming to discuss and close some of the gaps in the literature on models using options (and more generally coagents). Briefly surveying the theory behind these models, it also aims to provide a unifying point of view on the many diverse examples that fall under a same category called coagent network. Motivated by the result of [10] on parameter sharing of options, we revisit the theory of (a)synchronous Coagent Network [8] by generalizing the result to the context where parameters are shared among the function approximators of coagents. The proof is more intuitive and uses the concept of execution paths in a coagent network. Theoretically, this informs us of some necessary modifications to the algorithms found in the literature which make them more mathematically accurate. It also allows us to introduce a new simple option framework, Feedforward Option Network, which outperforms the previous option models in time to convergence and stability in the famous nonstationary Four Rooms task. In addition, a stabilization effect is observed in hierarchical models which justify the unnecessity of the target network in training such models. Finally, we publish our code which allows us to be flexible in our experiments settings.