HIGhER : Improving instruction following with Hindsight Generation for Experience Replay

TL;DR

The paper tackles the difficulty of grounding language instructions in reinforcement learning under sparse rewards by extending Hindsight Experience Replay to a language setting. It introduces Hindsight Generation for Experience Replay (HIGhER), which jointly learns a mapping from trajectories to substitute linguistic goals and a language-conditioned policy, relying only on environment signals and successful trajectories for supervision. Empirical results in BabyAI demonstrate that HIGhER closes much of the gap to an oracle-HER baseline, while providing robustness to noise and favorable sample efficiency for language grounding. The work highlights a practical, autonomous route to improve instruction-following without human demonstrations, with potential applicability to other goal modalities and future integration with additional exploration strategies.

Abstract

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay (HER) approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it complements other instruction following methods.

Figures (5)

• Figure 1: Upon positive trajectory, the agent trajectory is added to the RL replay buffer and the goal mapper dataset. Upon failed trajectory, the goal mapper is used to relabel the episode, and both trajectories are appended to the replay buffer. In the original HER paper, the mapping function is bypassed since they are dealing with spatial goals, and therefore, vanilla HER cannot be applied without external expert.
• Figure 2: Top: Agent performance with noisy mapping function. Bottom: Instruction generator accuracy over 5k pairs. Figures are averaged over 5 seeds and error bars shows one standard deviation.
• Figure 3: Left: learning curves for DQN, DQN+HER, DQN+HIGhER in a 10x10 gridworld with 10 objects with 4 attributes. The instruction generator is used after the vertical bar. Right: the mapping accuracy for the prediction of instructions. $m_{{\bm{w}}}$ starts being trained after collecting 1000 positive trajectories. Results are averaged over 5 seeds with one standard deviation.
• Figure 4: The instruction generator is triggered after collecting 0, 1000 and 2000 positive trajectories (i.e, approximately 0%, 20%, 50% accuracy). Even when the instruction generator is not accurate, the policy still makes steady progress and the final success rate is not impacted. Delaying the generator instructor does not provide additional benefit
• Figure 5: Transition distributions in the replay buffer between successful, unsuccessful and relabeled trajectories. We remove time-out trajectories for clarity, which accounts for $54\%$ of the transition in average ($\pm3\%$ over training).