Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills

TL;DR

Skill Set Optimization (SSO) is proposed for improving LLM actor performance through constructing and refining sets of transferable skills and performing in-context policy improvement.

Abstract

Large language models (LLMs) have recently been used for sequential decision making in interactive environments. However, leveraging environment reward signals for continual LLM actor improvement is not straightforward. We propose Skill Set Optimization (SSO) for improving LLM actor performance through constructing and refining sets of transferable skills. SSO constructs skills by extracting common subtrajectories with high rewards and generating subgoals and instructions to represent each skill. These skills are provided to the LLM actor in-context to reinforce behaviors with high rewards. Then, SSO further refines the skill set by pruning skills that do not continue to result in high rewards. We evaluate our method in the classic videogame NetHack and the text environment ScienceWorld to demonstrate SSO's ability to optimize a set of skills and perform in-context policy improvement. SSO outperforms baselines by 40% in our custom NetHack task and outperforms the previous state-of-the-art in ScienceWorld by 35%.

Figures (7)

• Figure 1: Example of a interactive text task and skill.
• Figure 2: Each iteration of SSO collects a trajectory of interactions with the current LLM actor, uses this trajectory to construct new skills and filter poorly performing skills, and updates the skill set for use in the next iteration. New skills are constructed by extracting, scoring, and sampling sets of similar subtrajectories that are then used to generate subgoals and instructions for skills. Skills are filtered based on the discounted future rewards observed after executing a skill.
• Figure 3: Comparison between in-context skills, fewshot trajectory examples, and no in-context information on the Melting Temperature ScienceWorld and NetHack tasks.
• Figure 4: We compare SSO with ReAct and Reflexion baselines in ScienceWorld and NetHack domains. We also compare with the previous state-of-the-art for ScienceWorld, CLIN. In ScienceWorld we evaluate adaptation---attempting a single variant five times---and transfer---learning on 10 train variants for 30 iterations before testing on the heldout test variants. In NetHack we test task adaptation across 30 iterations.
• Figure 5: Skill Set statistics for ScienceWorld and NetHack during training. Skill set size measures the number of skills created minus those that were pruned during refinement. Executed skills is the average number of unique skills executed in a trajectory as reported by the LLM actor. Finally, the average task score is reported throughout training.