STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order

TL;DR

STO-RL tackles offline RL for long-horizon, sparse-reward tasks by leveraging an LLM to generate temporally ordered subgoals and a state-to-subgoal mapping. It then applies a subgoal-temporal-order aware potential-based reward shaping to densify rewards and promote progress toward the final goal, while preserving optimality. The framework achieves faster convergence, higher success rates, and shorter trajectories across discrete and continuous benchmarks (CliffWalking, FourRoom, PointMaze-UMaze, PointMaze-Medium) compared to offline goal-conditioned and hierarchical baselines, and exhibits robustness to imperfect LLM subgoal sequences. This work demonstrates a practical, scalable approach that bridges high-level planning with low-level offline RL, enabling more reliable long-horizon policy learning without online interaction.

Abstract

Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing goal-conditioned and hierarchical offline RL methods decompose such tasks and generate intermediate rewards to mitigate limitations of traditional offline RL, but usually overlook temporal dependencies among subgoals and rely on imprecise reward shaping, leading to suboptimal policies. To address these issues, we propose STO-RL (Offline RL using LLM-Guided Subgoal Temporal Order), an offline RL framework that leverages large language models (LLMs) to generate temporally ordered subgoal sequences and corresponding state-to-subgoal-stage mappings. Using this temporal structure, STO-RL applies potential-based reward shaping to transform sparse terminal rewards into dense, temporally consistent signals, promoting subgoal progress while avoiding suboptimal solutions. The resulting augmented dataset with shaped rewards enables efficient offline training of high-performing policies. Evaluations on four discrete and continuous sparse-reward benchmarks demonstrate that STO-RL consistently outperforms state-of-the-art offline goal-conditioned and hierarchical RL baselines, achieving faster convergence, higher success rates, and shorter trajectories. Ablation studies further confirm STO-RL's robustness to imperfect or noisy LLM-generated subgoal sequences, demonstrating that LLM-guided subgoal temporal structures combined with theoretically grounded reward shaping provide a practical and scalable solution for long-horizon offline RL.

Figures (9)

• Figure 1: Overview of STO-RL. The agent first prompts an LLM to decompose task instructions into temporally ordered subgoals $\mathcal{G}=\{G_1, G_2, ..., G_K\}$ with a mapping function $h$ linking each state $s_t$ to its subgoal index $k_t$; then applies potential-based reward shaping to assign dense rewards to each transition $(s_t, a_t, s_{t+1})$ in dataset $D$ based on the mapping function $h$, followed by training an offline RL agent on the augmented dataset to learn the optimal policy.
• Figure 2: A toy example illustrating how the subgoal–temporal–order–aware PBRS penalizes longer successful trajectories. A virtual trajectory $\tau^3$ is constructed with the same length $T$ and discounted return as longer trajectory $\tau^2$ ($R(\tau^3)=R(\tau^2)$), consisting of the first $T-L$ steps sharing subgoal indices and return with shorter $\tau^1$, followed by $L$ non-progress transitions that yield negative returns under Theorem 2.
• Figure 3: Environments with finite, discrete action spaces.
• Figure 4: Learning curves for the CliffWalking and FourRoom tasks, smoothed using a moving average over 50 iterations.
• Figure 5: Value functions for the FourRoom task. ‘S’ denotes the start state, ‘G’ the goal state, and ‘W’ the walls. Arrows indicate the optimal actions under the learned policy.

Theorems & Definitions (5)

• proof
• proof
• proof
• proof
• proof