Adversarial Imitation Learning (AIL) is a class of popular state-of-the-art Imitation Learning algorithms commonly used in robotics. In AIL, an artificial adversary's misclassification is used as a reward signal that is optimized by any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings, the reward in AIL is $differentiable$ but current model-free RL algorithms do not make use of this property to train a policy. The reward is AIL is also shaped since it comes from an adversary. We leverage the differentiability property of the shaped AIL reward function and formulate a class of Actor Residual Critic (ARC) RL algorithms. ARC algorithms draw a parallel to the standard Actor-Critic (AC) algorithms in RL literature and uses a residual critic, $C$ function (instead of the standard $Q$ function) to approximate only the discounted future return (excluding the immediate reward). ARC algorithms have similar convergence properties as the standard AC algorithms with the additional advantage that the gradient through the immediate reward is exact. For the discrete (tabular) case with finite states, actions, and known dynamics, we prove that policy iteration with $C$ function converges to an optimal policy. In the continuous case with function approximation and unknown dynamics, we experimentally show that ARC aided AIL outperforms standard AIL in simulated continuous-control and real robotic manipulation tasks. ARC algorithms are simple to implement and can be incorporated into any existing AIL implementation with an AC algorithm. Video and link to code are available at: https://sites.google.com/view/actor-residual-critic.
翻译:ADIL (AIL) 是机器人通常使用的一种流行的、 最高级的智能学习算法。 在 AIL 中, 人为对手的错误分类被使用为一种奖励信号, 任何标准的SEAREEAR( RL) 算法都会优化。 与大多数 RL 设置不同, AIL 的奖赏是美元可差别的, 但目前没有模型的 RL 算法并不使用此属性来培训政策。 奖赏是 AIL 的简单自动算法。 我们利用了 自动智能学习学习算法的可变性属性, 并开发了一种ARC( ARC) 运算法的分类法。 ARC 算法与一个标准的ALA( ARC) 相匹配。 使用标准的ALA( ARC) 匹配算法, 并使用一个常规的 ALILA( ) 算法, 运行一个直径直径直径直径比的 ALLA( ) 。