Guiding Skill Discovery with Foundation Models

TL;DR

FoG introduces foundation-model guided skill discovery to align unsupervised reinforcement learning with human preferences. It extracts a score function from foundation models and uses it to reweight the intrinsic skill discovery rewards, effectively turning the score into a distance-like metric in the DSD objective. Across state-based and pixel-based tasks, FoG reduces undesirable behaviors and hazards while discovering diverse, human-aligned skills, often without expert demonstrations. The approach demonstrates robustness to score noise and generalizes across inputs, with ablations clarifying the roles of the score function and hyperparameters in shaping behavior.

Abstract

Learning diverse skills without hand-crafted reward functions could accelerate reinforcement learning in downstream tasks. However, existing skill discovery methods focus solely on maximizing the diversity of skills without considering human preferences, which leads to undesirable behaviors and possibly dangerous skills. For instance, a cheetah robot trained using previous methods learns to roll in all directions to maximize skill diversity, whereas we would prefer it to run without flipping or entering hazardous areas. In this work, we propose a Foundation model Guided (FoG) skill discovery method, which incorporates human intentions into skill discovery through foundation models. Specifically, FoG extracts a score function from foundation models to evaluate states based on human intentions, assigning higher values to desirable states and lower to undesirable ones. These scores are then used to re-weight the rewards of skill discovery algorithms. By optimizing the re-weighted skill discovery rewards, FoG successfully learns to eliminate undesirable behaviors, such as flipping or rolling, and to avoid hazardous areas in both state-based and pixel-based tasks. Interestingly, we show that FoG can discover skills involving behaviors that are difficult to define. Interactive visualisations are available from https://sites.google.com/view/submission-fog.

Figures (16)

• Figure 1: FoG leverages foundation models (such as ChatGPT, Claude and CLIP) to score states in relation to given commands during training. These scores are used to re-weight the rewards of the underlying skill discovery algorithm. Left: In state-based tasks (top row), task descriptions are provided to foundation models, which are queried to generate a score function $f(s)$ based on our requirements. In pixel-based tasks (bottom row), the current visual state, textual descriptions of desirable and undesirable intentions are input to foundation models to obtain embeddings. These embeddings are then used to form the score function $f(s)$, see \ref{['eq:fs']}. Right: During training, rewards of the underlying skill discovery method ($r_{skill}$) are re-weighted using the score function. Re-weighting $r_{skill}$ (we use METRA park2023metra) by the score function is equivalent with using the score function as the distance metric in the DSD objective.
• Figure 2: Environments used in our work. HalfCheetah and Ant are state-based while the other three are pixel-based.
• Figure 3: Comparison between METRA and FoG on state-based HalfCheetah and Ant. In both tasks, foundation models successfully capture the relevant state dimension and set threshold for it. Left: FoG learns not to roll in HalfCheetah, while METRA rolls over $50\%$ of the time, violating our intention. Right: FoG learns to not move to south in Ant, and METRA learns to move in all directions.
• Figure 4: Left: Executions of example skills from different agents in pixel-based environment, Cheetah. From top to bottom: METRA, METRA+, LSD, DoDont, DoDont+, FR-SAC, FoG. Right: Percentage of flips (which should be prevented based on the guidance) and state coverage for different agents. METRA, METRA+, DoDont, and DoDont+ discover diverse states but often flip. LSD and FR-SAC fail to learn diverse skills. FoG excels with high state coverage and minimal flipping.
• Figure 5: Top: Results on the pixel-based environment Cheetah, with learned skills shown in x-coordinates. METRA+ learns to perfectly avoid the undesirable area and FoG has a strong preference to go to the desirable area, as also clearly visible from the Safe State Coverage on the right. Other agents fail. Bottom: Results on the pixel-based environment Quadruped, with learned skills shown as xy-coordinates. Similar conclusions can be drawn regarding most of agents. Unlike in Cheetah, DoDont successfully learns to avoid the bottom-left areas.