SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding

TL;DR

This work introduces SCALAR, a bidirectional framework coupling LLM planning with RL through a learned skill library, and corrects LLM priors by analyzing RL trajectories and optionally saves environment states at skill boundaries to improve sample efficiency.

Abstract

LM-based agents excel when given high-level action APIs but struggle to ground language into low-level control. Prior work has LLMs generate skills or reward functions for RL, but these one-shot approaches lack feedback to correct specification errors. We introduce SCALAR, a bidirectional framework coupling LLM planning with RL through a learned skill library. The LLM proposes skills with preconditions and effects; RL trains policies for each skill and feeds back execution results to iteratively refine specifications, improving robustness to initial errors. Pivotal Trajectory Analysis corrects LLM priors by analyzing RL trajectories; Frontier Checkpointing optionally saves environment states at skill boundaries to improve sample efficiency. On Craftax, SCALAR achieves 88.2% diamond collection, a 1.9x improvement over the best baseline, and reaches the Gnomish Mines 9.1% of the time where prior methods fail entirely.

Figures (14)

• Figure 1: Overview of SCALAR's LLM-RL loop. The LLM reads the game manual and proposes candidate skills as symbolic operators with preconditions and effects (1). Skills whose preconditions are reachable from the current frontier are marked feasible and added to the skill graph; infeasible skills are deferred (2). Given a target skill, a planner sequences existing skills to reach a state satisfying its preconditions; an RL agent trains a policy for the target skill by executing these prerequisite skills and then learning from its own exploration (3). After training, trajectory analysis examines successful executions to verify or correct the LLM's initial specification of preconditions and resource requirements (4). Revised specifications update the skill graph and inform subsequent LLM proposals. This loop repeats until the target skill achieves sufficient success rate.
• Figure 2: Training curves on Craftax-Classic (left two panels) and Craftax (right three panels). X-axis: training frames in millions. Y-axis: success rate (%) over 256 evaluation episodes. Lines show mean over 5 seeds; shaded regions show standard error. SCALAR (black solid) and SCALAR-TRXL (blue solid) are our method; dashed lines are PPO variants; dotted lines are PQN variants.
• Figure 3: Resources collected before attempting diamond, comparing SCALAR with and without trajectory analysis. Bars show total wood, stone, and iron gathered. Left bars: LLM's initial estimates. Right bars: after trajectory analysis refines requirements.
• Figure 4: Fraction of training frames spent on the target skill (y-axis) for different reset probabilities $p \in \{0, 0.5, 0.9, 0.99\}$. Four panels: Craftax-Classic (Collect Diamond) and Craftax (Enter Dungeon, Kill 8 Orcs, Enter Gnomish Mines). Higher $p$ means more frequent resets to saved frontier states.
• Figure 5: Comparison of Craftax-Classic and Craftax environments. Craftax-Classic is shown on the left; Craftax on the right.