Transformer has achieved great successes in learning vision and language representation, which is general across various downstream tasks. In visual control, learning transferable state representation that can transfer between different control tasks is important to reduce the training sample size. However, porting Transformer to sample-efficient visual control remains a challenging and unsolved problem. To this end, we propose a novel Control Transformer (CtrlFormer), possessing many appealing benefits that prior arts do not have. Firstly, CtrlFormer jointly learns self-attention mechanisms between visual tokens and policy tokens among different control tasks, where multitask representation can be learned and transferred without catastrophic forgetting. Secondly, we carefully design a contrastive reinforcement learning paradigm to train CtrlFormer, enabling it to achieve high sample efficiency, which is important in control problems. For example, in the DMControl benchmark, unlike recent advanced methods that failed by producing a zero score in the "Cartpole" task after transfer learning with 100k samples, CtrlFormer can achieve a state-of-the-art score with only 100k samples while maintaining the performance of previous tasks. The code and models are released in our project homepage.
翻译:变异器在学习视觉和语言代表方面取得了巨大的成功,这在各种下游任务中是普遍的。在视觉控制中,学习可在不同控制任务之间转移的可转移国家代表对于减少培训样本规模很重要。然而,将变异器移植到样本高效视觉控制仍然是一个棘手和未解决的问题。为此,我们提议了一个新的控制变异器(CtrlFormer),拥有许多前艺术所不具备的吸引人的好处。首先,CtrlFormer联合学习了不同控制任务之间视觉象征物和政策象征之间的自我注意机制,在这种系统中,多任务代表物可以学习和转移,而不会被灾难性地遗忘。第二,我们仔细设计了一个对比强化学习模式来培训CtrlFormer,使其能够实现高样本效率,这对于控制问题非常重要。例如,在DM控制基准中,与最近的先进方法不同,在以100公里样本传输学习后,在“Cartpole”任务中产生了零分,CtrFormer可以在维持前项任务执行过程时,只有100公里样本才能获得州分。