The classic Reinforcement Learning (RL) formulation concerns the maximization of a scalar reward function. More recently, convex RL has been introduced to extend the RL formulation to all the objectives that are convex functions of the state distribution induced by a policy. Notably, convex RL covers several relevant applications that do not fall into the scalar formulation, including imitation learning, risk-averse RL, and pure exploration. In classic RL, it is common to optimize an infinite trials objective, which accounts for the state distribution instead of the empirical state visitation frequencies, even though the actual number of trajectories is always finite in practice. This is theoretically sound since the infinite trials and finite trials objectives can be proved to coincide and thus lead to the same optimal policy. In this paper, we show that this hidden assumption does not hold in the convex RL setting. In particular, we show that erroneously optimizing the infinite trials objective in place of the actual finite trials one, as it is usually done, can lead to a significant approximation error. Since the finite trials setting is the default in both simulated and real-world RL, we believe shedding light on this issue will lead to better approaches and methodologies for convex RL, impacting relevant research areas such as imitation learning, risk-averse RL, and pure exploration among others.
翻译:经典强化学习(RL) 的提法涉及最大程度的升级奖励功能。 最近,引入了Convex RL, 将RL的提法扩大到所有由政策引发的州分配的统函数的所有目标。 值得注意的是, convex RL 包含一些不属于标度配制的相关应用, 包括模仿学习、 风险规避RL 和纯粹探索。 在经典的RL 中, 常见的是优化一个无限的试验目标, 这个目标说明国家的分配情况, 而不是经验性国家访问频率, 尽管实际的轨迹数量在实践上总是有限。 这在理论上是有道理的, 因为无限的试验和有限试验目标可以被证明是同步的, 从而导致相同的最佳政策。 在本文中,我们表明这一隐蔽的假设不能在Convex RL 设置中存在。 特别是, 我们证明错误地优化了无限的试验目标, 取代实际的有限试验, 通常会导致重大的近似错误。 由于这一有限的审判是在模拟和真实的RxL 研究方法中, 我们相信, 将这种有限的试验设定在模拟和真实的Rx 学习风险 。