Reinforced Imitation in Heterogeneous Action Space

TL;DR

This work addresses learning policies when the expert’s actions are unavailable and the learner’s action space differs, by integrating sparse environmental rewards with state-based imitation signals. The authors extend GAIL into Reinforced Imitation Learning from Observations (RILO), featuring a state-pair discriminator, a mixture reward r_t = r^{env}_t + λ r^{imt}_t, and a self-exploration mechanism that gradually shifts emphasis from imitation to environment rewards. Through systematic grid-world and ViZDoom experiments, RILO demonstrates that agents can outperform pure imitation or reinforcement learning, especially when action spaces are heterogeneous, and that self-exploration enhances learning stability and final performance. The results highlight the practical potential of leveraging sparse rewards and state-only demonstrations for robust policy learning in diverse morphologies and partially observable settings.

Abstract

Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent and an expert use different actions from each other. We assume that the agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. In addition, this method adapts the agent's policy based on either mimicking expert behavior or maximizing sparse reward. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when its actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert, due to the optimized usage of sparse rewards.

Figures (10)

• Figure 1: A representation of RILO framework. An agent (learner) with a policy $\pi_\theta$ interacts with an environment producing a trajectory of states and a sparse reward. Imitation rewards are acquired by comparing state-only observations from the learner and the expert. The policy is updated with a combination of both rewards.
• Figure 2: (a-b) Example of an initial state of the environment with blue and red squares representing the goal and the agent, respectively. (a) In P grid world (a), with the $5\times5$ visible area around the agent highlighted. In F map (b), the agent is initially placed in one of the locations covered by green triangle. (c-f) Set of actions for different agents considered in our experiments. Green squares represent the possible locations after the move.
• Figure 3: Comparison of RILO experiments with different imitation rewards and the effect of self-exploration on different expert-learner pairs. In the first row results for F map are presented (a-d) while results for P grid world are shown in the second row (e-h). Each chart represents how a given expert aids each learner, i.e. learner move styles vary while expert is fixed. We show results in which agents use self-exploration (right, dark) or not (left, bright). Additionally, the relation between the learner and the expert is coded using one of four symbols: the same action spaces (=), disjoint action spaces (D), superior learner (L), or superior expert (E).
• Figure 4: Performance of algorithms (success rate) with respect to the number of iterations (in millions), assuming different learners. In all cases, the expert is the 4-way agent. See text for details and Appendix \ref{['app:plots']} for higher resolution figures or other experts results.
• Figure 5: Comparison of RILO experiments with different imitation rewards and the effect of self-exploration for different learners. Expert is single in all cases, hence the first set of experiments (single) is with agent having the same action space, agent is superior for the middle one (multi). The largest difference between action spaces is for the last case (right).