We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs). In this setting, both the reward function and the transition kernel are linear with respect to the given feature maps and are allowed to vary over time, as long as their respective parameter variations do not exceed certain variation budgets. We propose the $\underline{\text{p}}$eriodically $\underline{\text{r}}$estarted $\underline{\text{o}}$ptimistic $\underline{\text{p}}$olicy $\underline{\text{o}}$ptimization algorithm (PROPO), which is an optimistic policy optimization algorithm with linear function approximation. PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement, which are tailored for policy optimization in a non-stationary environment. In addition, only utilizing the technique of sliding window, we propose a value-iteration algorithm. We establish dynamic upper bounds for the proposed methods and a matching minimax lower bound which shows the (near-) optimality of the proposed methods. To our best knowledge, PROPO is the first provably efficient policy optimization algorithm that handles non-stationarity.
翻译:在非静止线性线性内核Markov 决策程序(MDPs)中,我们研究侧侧强化学习(RL) 。在这种环境下,奖励功能和过渡内核对于特定特效地图都是线性,只要各自的参数变异不超过某些变异预算,允许随时间变化而变化。我们提议在非静止环境中为政策优化量身定制的$underline_text{prodeline}untext{o}$popimistic $_underline{producy $_underline{pline{prode_$underline}(PROPO)算法(PRO),这是一种乐观的政策优化算法,与线性函数相近。 PROPO有两个机制:基于滑动窗口的政策评价和定期启动政策改进。此外,我们只利用滑动窗口技术,我们建议一种增值算法。我们为拟议的方法建立动态的上限,并配对最低的缩式缩式缩式算法,它首先显示(最接近的)最优化政策。