Learning Diverse Policies with Soft Self-Generated Guidance

TL;DR

The paper tackles reinforcement learning under sparse and deceptive rewards by using diverse, self-generated past trajectories as soft guidance rather than imitation. It proposes POSE, a two-step framework that combines policy improvement with soft self-imitation and a diversity-driven exploration strategy across a team of agents, regulated by a trajectory-space distance constraint based on Maximum Mean Discrepancy. A novel diversity metric and a Lagrangian-based optimization guide policy updates to revisit promising regions while expanding into underexplored areas, improving exploration and avoiding local optima. Experimental results on large grid-world mazes and MuJoCo benchmarks show that POSE outperforms strong baselines in both exploration and learning speed, demonstrating practical impact for long-horizon, sparse-reward tasks. Key mathematical components include the distance dist(τ, μ) = $D_{MMD}\big(b(τ), b(μ)\big)$ and the constrained objective L(θ) = J(θ) − $\sigma E_{τ\in B^i}[d(τ, M^i)]$, with policy updates guided by $\nabla_θL = \nabla_θJ − σ E_{τ\in B^i}[d(τ, M^i) \nabla_θ \log p_θ(τ)]$, plus the diversity objective div(Π) based on mean trajectories and $D_{MMD}$ between agents.

Abstract

Reinforcement learning (RL) with sparse and deceptive rewards is challenging because non-zero rewards are rarely obtained. Hence, the gradient calculated by the agent can be stochastic and without valid information. Recent studies that utilize memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods often require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This paper develops an approach that uses diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm combines a policy improvement step with an additional exploration step using offline demonstration data. The main contribution of this paper is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories while still being able to learn without rewards and approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the team's diversity and regulate exploration. The proposed algorithm is evaluated on discrete and continuous control tasks with sparse and deceptive rewards. Compared with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and avoiding local optima.

Figures (8)

• Figure 1: The framework of POSE. The diverse exploration in POSE leverages multiple agents to sample training batches and use the measurement of diversity to encourage agents to collect diverse trajectories. In contrast, the traditional RL, e.g., PPO schulman2017proximal or SAC haarnoja2018soft, uses a single agent to collect data. In the meantime, every agent maintains a replay buffer and stores specific trajectories in this buffer. These past good trajectories can guide the agents to revisit the region where the agents can obtain rewards with a higher probability.
• Figure 2: A collection of environments with discrete state-action spaces that we use. (a) Huge grid-world maze with sparse rewards: Key-Door-Treasure domain. The agent should pick up the key (+2) in the right-down room in order to open the blue door (+4) and collect the treasure (+4) in the middle-up room to maximize the reward. (b) Huge grid-world maze with deceptive rewards. There is an apple in the left-up room that gives small rewards (+2) and a treasure in the middle-up room which generates the higher rewards (+10).
• Figure 3: A collection of environments with continuous state-action spaces that we use: (a) Swimmer in the maze, (b) Ant in the maze.
• Figure 4: Learning curves of average return and success rate in the huge grid-world maze with sparse rewards. Specifically, the success rate is used to illustrate the frequency at which agents reach the globally optimal goal during the training process.
• Figure 5: Learning curves of average return and success rate in the huge grid-world maze with deceptive rewards. Specifically, the success rate is used to illustrate the frequency at which agents reach the globally optimal goal during the training process.