Learning Hierarchical Procedural Memory for LLM Agents through Bayesian Selection and Contrastive Refinement

TL;DR

MACLA presents a memory-augmented framework that decouples reasoning from learning by maintaining a frozen LLM and updating an external hierarchical procedural memory. Three core mechanisms—Bayesian procedure selection, contrastive refinement, and meta-procedural composition—enable sample-efficient, interpretable, and continual improvement without LLM parameter updates. Across four diverse benchmarks, MACLA achieves state-of-the-art average performance with a 7B model and delivers dramatic reductions in memory-construction time while compressing thousands of trajectories into hundreds of reusable procedures. The work demonstrates robust generalization, efficient inference, and a principled approach to managing procedural knowledge for long-horizon tasks.

Abstract

We present MACLA, a framework that decouples reasoning from learning by maintaining a frozen large language model while performing all adaptation in an external hierarchical procedural memory. MACLA extracts reusable procedures from trajectories, tracks reliability via Bayesian posteriors, selects actions through expected-utility scoring, and refines procedures by contrasting successes and failures. Across four benchmarks (ALFWorld, WebShop, TravelPlanner, InterCodeSQL), MACLA achieves 78.1 percent average performance, outperforming all baselines. On ALFWorld unseen tasks, MACLA reaches 90.3 percent with 3.1 percent positive generalization. The system constructs memory in 56 seconds, 2800 times faster than the state-of-the-art LLM parameter-training baseline, compressing 2851 trajectories into 187 procedures. Experimental results demonstrate that structured external memory with Bayesian selection and contrastive refinement enables sample-efficient, interpretable, and continually improving agents without LLM parameter updates.

Figures (7)

• Figure 1: Comparison between existing LLM-based trajectory learning (top) and the proposed memory-augmented contrastive learning agent (MACLA, bottom). Existing methods train trajectories $(T, A, O, R)$ (Task, Action, Observation, Reward) into LLM parameters through post-training (finetuning and/or RLHF), whereas MACLA constructs procedural and meta-procedural memory externally through frozen LLM abstraction, segmentation, Bayesian selection, and contrastive refinement. Memories are learned during memory construction. Besides learning during memory construction, MACLA enables inference-time learning in which outputs are verified in the task environment, with feedback used for contrastive refinement on the retrieved memories. Meta-procedural learning enables the composition policy to be learned among procedures.
• Figure 2: Ablation study varying maximum procedural memory capacity. (a) Success rate on ALFWorld seen/unseen splits saturates beyond 150 procedures, with diminishing returns from 150→200 (+1.6% unseen) and slight decline at 300 (-0.2%). (b) Average Bayesian posterior $\frac{\alpha}{\alpha + \beta}$ plateaus at 0.79, showing extra capacity adds redundancy rather than quality.
• Figure 3: Bayesian learning dynamics for top-5 procedures during 200 test episodes. (a) Cumulative success count $\alpha$ grows at different rates: Navigate (blue) reaches 150+ invocations, while task-specific procedures (Heat/Cool, green/red) accumulate evidence more slowly due to limited applicability. (b) Posterior success rates $\frac{\alpha}{\alpha + \beta}$ converge above 0.75 within 50 episodes, with variance decreasing as $O(1/(\alpha{+}\beta))$.
• Figure 4: Analysis of 200+ pruned procedures during ALFWorld training. (a) Bimodal success rate distribution: pruned procedures (red, mean 0.42) separate cleanly from retained procedures (green, mean 0.79), validating utility-based retention. (b) Scatter plot shows pruned procedures cluster in bottom-left (young + rarely used), with no high-quality procedures (>0.7 success, >10 uses) pruned.
• Figure 5: Cross-domain analysis. (a) Memory reuse: 51% (SQL) to 78% (ALFWorld). (b) Procedure reliability: 64% (SQL) to 81% (ALFWorld). (c) Meta-procedure usage: 18% (SQL) to 51% (TravelPlanner).