Most real-world optimization problems have multiple objectives. A system designer needs to find a policy that trades off these objectives to reach a desired operating point. This problem has been studied extensively in the setting of known objective functions. We consider a more practical but challenging setting of unknown objective functions. In industry, this problem is mostly approached with online A/B testing, which is often costly and inefficient. As an alternative, we propose interactive multi-objective off-policy optimization (IMO^3). The key idea in our approach is to interact with a system designer using policies evaluated in an off-policy fashion to uncover which policy maximizes her unknown utility function. We theoretically show that IMO^3 identifies a near-optimal policy with high probability, depending on the amount of feedback from the designer and training data for off-policy estimation. We demonstrate its effectiveness empirically on multiple multi-objective optimization problems.
翻译:多数现实世界优化问题有多重目标。 系统设计师需要找到一种政策,将这些目标转换成一个理想的操作点。 在设定已知的客观功能时,已经对该问题进行了广泛的研究。 我们考虑的是更实际但富有挑战性的未知目标功能设置。 在行业中,这个问题大多通过在线A/B测试来解决,而在线A/B测试往往成本高、效率低。 作为替代办法,我们提出了互动式的多目标脱政策优化(IMO3/3)。 我们的方法中的关键想法是利用以非政策方式评估的政策与系统设计师互动,以发现哪些政策最大限度地发挥她未知的实用功能。 我们理论上表明,IMO3确定了一种近乎最佳的政策,其可能性很大,取决于设计师的反馈量和对非政策估算的培训数据。 我们从经验上证明了它对于多种多目标优化问题的有效性。