Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization. Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the generated tasks. This is a non-stationary process where the task distribution evolves along with agent policies; creating an instability over time. While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches. To this end, we introduce CLUTR: a novel unsupervised curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization. It first trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold. Next, a teacher agent creates a curriculum by maximizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold. Using the fixed-pretrained task manifold, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability. Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in the challenging CarRacing and navigation environments: achieving 10.6X and 45\% improvement in zero-shot generalization, respectively. CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, while requiring 500X fewer environment interactions.
翻译:强化学习(RL) 算法通常以抽样效率低下和难以概括而闻名。 最近,无监管环境设计(UED)通过同时学习任务分配和代理政策来学习对所产生任务进行任务分配和代理政策,成为零点概括的新范式。 这是一个非静止的过程,任务分配随着代理政策而演变;随着时间推移造成不稳定。 虽然过去的工作展示了这种方法的潜力,但从任务空间有效取样仍然是开放的挑战,阻碍了这些方法。为此,我们引入了CLUTR:一种创新的、不受监管的学习算法,将任务表达和课程学习分为两阶段。它首先对随机生成的任务进行经常性的自动变异编码,以学习潜在的任务组合。接下来,教师代理人通过最大限度地实现基于小型任务REGRETRET的一组潜在任务目标来创建课程。我们通过固定的任务多重,我们显示CLUTR成功地克服了非固定性的问题,提高了课程的稳定性。 我们的实验结果显示CLUTR在C-C-CARED 10级环境中需要C- CROD- CROD-C- CROD-C-C- dal-C- drovical- drovical- drodustral- droutal- 10- 和PAD-C- drofal- drofal- 10- drocual- disal- droutal- disal- disal- disal- disal- disal- 10- drocument- drocumental- drocumental- drocumental- drocumental-C- 和PAD- 10-C-C-C-C-C-C-C- disal- disal-C-C-C-C-C-C-C-C- 10-C-C-C-C-C-C-C-I) 和C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-C-I-I-C-R-I-I-C-C-C-C-I</s>