Reinforcement learning (RL) involves performing exploratory actions in an unknown system. This can place a learning agent in dangerous and potentially catastrophic system states. Current approaches for tackling safe learning in RL simultaneously trade-off safe exploration and task fulfillment. In this paper, we introduce a new generation of RL solvers that learn to minimise safety violations while maximising the task reward to the extent that can be tolerated by the safe policy. Our approach introduces a novel two-player framework for safe RL called Distributive Exploration Safety Training Algorithm (DESTA). The core of DESTA is a game between two adaptive agents: Safety Agent that is delegated the task of minimising safety violations and Task Agent whose goal is to maximise the environment reward. Specifically, Safety Agent can selectively take control of the system at any given point to prevent safety violations while Task Agent is free to execute its policy at any other states. This framework enables Safety Agent to learn to take actions at certain states that minimise future safety violations, both during training and testing time, while Task Agent performs actions that maximise the task performance everywhere else. Theoretically, we prove that DESTA converges to stable points enabling safety violations of pretrained policies to be minimised. Empirically, we show DESTA's ability to augment the safety of existing policies and secondly, construct safe RL policies when the Task Agent and Safety Agent are trained concurrently. We demonstrate DESTA's superior performance against leading RL methods in Lunar Lander and Frozen Lake from OpenAI gym.
翻译:强化学习( RL) 是指在一个未知的系统中进行探索性活动。 这可以在危险的和潜在的灾难性系统状态中放置一个学习代理。 目前在RL中处理安全学习的当前方法是同时进行交易安全勘探和完成任务。 在本文件中, 我们引入新一代的RL解答器, 学会尽量减少违反安全规定的行为, 同时在安全政策允许的范围内最大限度地扩大任务奖励。 我们的方法为安全RL引入了一个新型的双人框架, 称为分配探索安全培训等级( DESTA ) 。 DESTA的核心是两个适应性的代理器之间的一个游戏: 安全代理器, 负责尽量减少违反安全的情况, 任务执行者的任务是最大限度地增加环境奖励。 具体地说, 安全代理器可以在任何特定地点有选择地控制系统, 防止违反安全规定的行为, 而任务执行者可以自由地在任何其他州执行其政策。 这个框架让安全代理器学会在某些州采取行动, 尽量减少未来违反安全规定的行为, 在培训和测试期间, 任务代理器在任何地方执行任务表现最大程度的任务。 理论上, 我们证明, DESA 的高级操作能力 显示我们最稳定地展示了REA 的违反安全政策。</s>