ExperienceWeaver: Optimizing Small-sample Experience Learning for LLM-based Clinical Text Improvement

TL;DR

ExperienceWeaver introduces a hierarchical, training‑free framework that distills multi‑dimensional clinician feedback into structured experience, then injects this experience via an agentic revision pipeline to improve LLM‑based clinical text editing in small‑sample settings. By separating experience distillation (Stage 1) from retrieval (Stage 2) and coupling it with an orchestrated agent system and multi‑dimensional feedback, the method learns how to revise rather than merely what to revise. Across four clinical datasets, ExperienceWeaver yields consistent improvements over strong baselines and several SOTA models, demonstrating strong performance in low‑data contexts and real‑world validation on error detection. The work highlights the practical value of structured, layered experience for reliable, domain‑focused text improvement with potential for broader adoption in clinical documentation workflows.

Abstract

Clinical text improvement is vital for healthcare efficiency but remains difficult due to limited high-quality data and the complex constraints of medical documentation. While Large Language Models (LLMs) show promise, current approaches struggle in small-sample settings: supervised fine-tuning is data-intensive and costly, while retrieval-augmented generation often provides superficial corrections without capturing the reasoning behind revisions. To address these limitations, we propose ExperienceWeaver, a hierarchical framework that shifts the focus from data retrieval to experience learning. Instead of simply recalling past examples, ExperienceWeaver distills noisy, multi-dimensional feedback into structured, actionable knowledge. Specifically, error-specific Tips and high-level Strategies. By injecting this distilled experience into an agentic pipeline, the model learns "how to revise" rather than just "what to revise". Extensive evaluations across four clinical datasets demonstrate that ExperienceWeaver consistently improves performance, surpassing state-of-the-art models such as Gemini-3 Pro in small-sample settings.

Figures (11)

• Figure 1: Project Motivation. Improving clinical text with evolving agents relies heavily on the efficiency of in‑context learning. Current RAG‑based methods often fail to provide sufficiently dense information and introduce redundancy in memory storage. The proposed method addresses these limitations by weaving experience from stored feedback, enabling more effective reasoning within memory.
• Figure 2: The pipeline of proposed ExperienceWeaver. The pipeline consists of two stages of weaving: first, a hierarchical auto‑combination and distillation for experience abstracted from feedback; second, a re‑weaving step to formulate error‑specific tips and functional strategy layers. The goal is to inject distilled multi-layered experience into the three phases in the agentic pipeline: error detection, text revision, and self‑critique.
• Figure 3: Sample contexts formulated by ExperienceWeaver.
• Figure 4: Performance Rank of base models as LLM-as-a-Judge.
• Figure 5: Case Study to explain how experience improves detection accuracy.