Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.
翻译:在智能决策系统的核心,如何代表政策和优化政策是一个根本问题。这个问题的根本挑战是政策空间的庞大和高度复杂,这加剧了政策学习的难度,特别是在现实世界的情景中。为了建立一个理想的替代政策空间,最近低维潜伏空间的政策代表性显示了其在改进评估和优化政策方面的潜力。这些研究涉及的关键问题是,我们应该以什么标准来抽取所希望的压缩和概括化的政策空间。然而,文献对政策抽象理论和政策代表性学习方法的研究较少。在这项工作中,我们首先努力填补空缺。首先,我们提出了一个统一的政策抽象理论,其中包括与不同层次的政策特征相关的三种政策抽象性。然后,我们将其归纳为三个政策指标,以量化政策的距离(即相似性),以便更方便地利用政策代表性。此外,我们建议一种政策代表性学习方法以深入的计量学习为基础。对于一些经验研究来说,我们首先要努力填补空缺。首先,我们提出了一种统一的政策抽象的理论抽象性理论理论理论,然后将我们提出的政策优化政策评价的精细度和方向性都包含我们政策评估的精细度。我们进行的政策格式化的精细度分析,在政策上,在进行政策上的精确度评估时,在评估时,在分析时,要显示我们的政策的精细政策演变中表现的精度和区域评价的精细度上是分度上的精细度上的精细度上是分度。