Seeing Differently, Acting Similarly: Imitation Learning with Heterogeneous Observations

In many real-world imitation learning tasks, the demonstrator and the learner have to act under different but full observation spaces. This situation generates significant obstacles for existing imitation learning approaches to work. Previous related works need to assume the coexistence of two observation spaces in the demonstrations or that all along the learning process. While in reality, the expert usually provides the demonstration with their observations only, and the observation coexistence will be limited due to the high cost. So in this work, we model the observation mismatch in the imitation learning problem with the above two challenges as a two-phase learning process, namely Heterogeneously Observable Imitation Learning (HOIL). We analyze the underlying learning issues with these challenges, i.e., the dynamics mismatch and the support mismatch, and further propose the Importance Weighting with REjection (IWRE) algorithm based on the techniques of importance-weighting and learning with rejection for querying to solve these issues across the observation spaces. Experimental results show that IWRE can successfully solve the difficult HOIL tasks, including the challenging task of transforming the vision-based demonstrations to random access memory (RAM)-based policies under the Atari domain.