Reinforcement learning (RL) is applied in a wide variety of fields. RL enables agents to learn tasks autonomously by interacting with the environment. The more critical the tasks are, the higher the demand for the robustness of the RL systems. Causal RL combines RL and causal inference to make RL more robust. Causal RL agents use a causal representation to capture the invariant causal mechanisms that can be transferred from one task to another. Currently, there is limited research in Causal RL, and existing solutions are usually not complete or feasible for real-world applications. In this work, we propose CausalCF, the first complete Causal RL solution incorporating ideas from Causal Curiosity and CoPhy. Causal Curiosity provides an approach for using interventions, and CoPhy is modified to enable the RL agent to perform counterfactuals. We apply CausalCF to complex robotic tasks and show that it improves the RL agent's robustness using a realistic simulation environment called CausalWorld.
翻译:强化学习( RL) 应用在广泛的领域。 RL 使代理商能够通过与环境互动自主学习任务。 更为关键的任务是,对RL系统稳健性的需求越高。 Causal RL 组合RL 和因果推断使RL更强。 Causal RL 代理商使用因果表示来捕捉从一个任务转到另一个任务的因果机制。 目前, Causal RL 的研究有限, 现有解决方案通常不完全或可行, 用于现实世界应用。 在这项工作中, 我们提出CausalCF, 第一个完整的 Causal RL 解决方案, 包括Causal Curioity 和 CoPhy。 Causal Curiosity 提供了使用干预的方法, 并且对 CoPhy 进行了修改, 以使 RL 代理商能够执行反事实。 我们用 Causal CF 来执行复杂的机器人任务, 并表明它能用一个称为 Causal World 的现实模拟环境来提高RL 代理商的稳健性。