Credit assignment is a fundamental problem in reinforcement learning, the problem of measuring an action's influence on future rewards. Improvements in credit assignment methods have the potential to boost the performance of RL algorithms on many tasks, but thus far have not seen widespread adoption. Recently, a family of methods called Hindsight Credit Assignment (HCA) was proposed, which explicitly assign credit to actions in hindsight based on the probability of the action having led to an observed outcome. This approach is appealing as a means to more efficient data usage, but remains a largely theoretical idea applicable to a limited set of tabular RL tasks, and it is unclear how to extend HCA to Deep RL environments. In this work, we explore the use of HCA-style credit in a deep RL context. We first describe the limitations of existing HCA algorithms in deep RL, then propose several theoretically-justified modifications to overcome them. Based on this exploration, we present a new algorithm, Credit-Constrained Advantage Actor-Critic (C2A2C), which ignores policy updates for actions which don't affect future outcomes based on credit in hindsight, while updating the policy as normal for those that do. We find that C2A2C outperforms Advantage Actor-Critic (A2C) on the Arcade Learning Environment (ALE) benchmark, showing broad improvements over A2C and motivating further work on credit-constrained update rules for deep RL methods.
翻译:信用分配是强化学习的根本问题,是衡量一项行动对未来奖励的影响的问题。信用分配方法的改进有可能提高RL算法在许多任务中的绩效,但迄今为止还没有被广泛采用。最近,提出了一系列名为Hindsight信用分配(HCA)的方法,根据行动可能导致观察到的结果,明确将信用分配给事后观察行动。这一方法具有提高数据使用效率的吸引力,但在很大程度上仍然是适用于一套有限的列表RL任务的理论性想法,不清楚如何将HCA推广到深RL环境。在这项工作中,我们探索在深度RL背景下使用HCA式信贷。我们首先描述现有的HCA算法在深RL中的局限性,然后提出若干理论上合理的修改,以克服这些局限性。根据这一探索,我们提出了一个新的算法,即信用-受约束的Advantage Actor-Ctic(C2A2A2A2C),它忽略了将HCAC式的更新政策如何影响未来成果,而我们在ADRA2C级标准上,在Sloudal Axim Aral Reformain Referal Acal Acal Acal Acal Acal 上, laut the slaview Acal be lax lax lax lax lax be lax lax lax la be lax lax lax lax la la la la la lax lax 作出进一步的改进的改进了这些改进了这些改进了这些改进了这些会计法,同时,我们的改进了这些会计法。