Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMs

Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai

TL;DR

SEAM introduces a lightweight Structured Experience Adapter Module that encodes instance-specific, utility-focused guidance into its parameters to steer a frozen LLM executor, avoiding external retrieval. Trained with a three-step forward-learning pipeline and GRPO, SEAM samples candidate guidance, evaluates it via executor rollouts, and updates to maximize downstream success, with optional deployment-time SFT. Across math-reasoning benchmarks, SEAM yields consistent accuracy gains over direct executor training and RAG-based baselines while maintaining low latency, and analyses show improved efficiency, generalization, and scalable experience benefits. The approach supports executor-specific, plug-in adaptation and highlights a practical path toward continual improvement of static LLMs without modifying the underlying solver. Limitations include domain transfer, cross-system adaptability, continual learning under distribution shifts, and safety considerations for deployment.

Abstract

Large language models (LLMs) are largely static and often redo reasoning or repeat mistakes. Prior experience reuse typically relies on external retrieval, which is similarity-based, can introduce noise, and adds latency. We introduce SEAM (Structured Experience Adapter Module), a lightweight, executor-specific plug-in that stores experience in its parameters and generates a structured, instance-tailored experience entry in a single forward pass to guide a frozen LLM executor. SEAM is trained for utility via executor rollouts and GRPO while keeping the executor frozen, and it can be further improved after deployment with supervised fine-tuning on logged successful trajectories. Experiments on mathematical reasoning benchmarks show consistent accuracy gains across executors with low overhead. Extensive ablations and analyses further elucidate the mechanisms underlying SEAM's effectiveness and robustness.

Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMs

TL;DR

SEAM introduces a lightweight Structured Experience Adapter Module that encodes instance-specific, utility-focused guidance into its parameters to steer a frozen LLM executor, avoiding external retrieval. Trained with a three-step forward-learning pipeline and GRPO, SEAM samples candidate guidance, evaluates it via executor rollouts, and updates to maximize downstream success, with optional deployment-time SFT. Across math-reasoning benchmarks, SEAM yields consistent accuracy gains over direct executor training and RAG-based baselines while maintaining low latency, and analyses show improved efficiency, generalization, and scalable experience benefits. The approach supports executor-specific, plug-in adaptation and highlights a practical path toward continual improvement of static LLMs without modifying the underlying solver. Limitations include domain transfer, cross-system adaptability, continual learning under distribution shifts, and safety considerations for deployment.

Abstract

Large language models (LLMs) are largely static and often redo reasoning or repeat mistakes. Prior experience reuse typically relies on external retrieval, which is similarity-based, can introduce noise, and adds latency. We introduce SEAM (Structured Experience Adapter Module), a lightweight, executor-specific plug-in that stores experience in its parameters and generates a structured, instance-tailored experience entry in a single forward pass to guide a frozen LLM executor. SEAM is trained for utility via executor rollouts and GRPO while keeping the executor frozen, and it can be further improved after deployment with supervised fine-tuning on logged successful trajectories. Experiments on mathematical reasoning benchmarks show consistent accuracy gains across executors with low overhead. Extensive ablations and analyses further elucidate the mechanisms underlying SEAM's effectiveness and robustness.
Paper Structure (61 sections, 16 equations, 5 figures, 8 tables)

This paper contains 61 sections, 16 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning from Experience.
  4. LLM and Reinforcement Learning.
  5. Method
  6. Problem Formulation
  7. Architecture
  8. Training of SEAM
  9. Step 1: Forward exploration (candidate generation).
  10. Step 2: Rollout-based evaluation (guidance utility).
  11. Step 3: Parametric library evolution (GRPO update).
  12. Optional: Deployment-time evolution via logged SFT.
  13. Experiment
  14. Experiment Settings
  15. Datasets.
  16. ...and 46 more sections

Figures (5)

  • Figure 1: Illustration of SEAM training and inference pipelines.
  • Figure 2: The cross-domain results for SEAM with Qwen3-4B as executor.
  • Figure 3: The results of scaling with experience size.
  • Figure 4: Input prompt for SEAM.
  • Figure 5: Input prompts for different executors.