Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Generalization Gap in LLM Planning: Tests and Verifier-Reward RL

Valerio Belcamino, Nicholas Attolino, Alessio Capitanelli, Fulvio Mastrogiovanni

TL;DR

This work investigates whether fine-tuned LLMs acquire transferable planning competence across PDDL-compatible domains or merely domain-specific patterns. It fine-tunes a 1.7B model on 40,000 domain–problem–plan tuples from IPC 2023 and evaluates in-domain and cross-domain generalization, revealing a strong in-domain performance (82.9% valid plans) but a complete generalization failure on unseen domains. Through three diagnostic variants—symbol anonymization, compact plan encoding, and verifier-reward RL—the study shows reliance on surface representations and domain-specific regularities, with limited gains in transfer despite improved training efficiency using VAL-based rewards. The work provides a multi-domain benchmark, diagnostic tools, and a VAL-based RL framework to probe and improve generalization in LLM-based planning, highlighting a persistent generalization gap and guiding future directions toward more abstract planning competence.

Abstract

Recent work shows that fine-tuned Large Language Models (LLMs) can achieve high valid plan rates on PDDL planning tasks. However, it remains unclear whether this reflects transferable planning competence or domain-specific memorization. In this work, we fine-tune a 1.7B-parameter LLM on 40,000 domain-problem-plan tuples from 10 IPC 2023 domains, and evaluate both in-domain and cross-domain generalization. While the model reaches 82.9% valid plan rate in in-domain conditions, it achieves 0% on two unseen domains. To analyze this failure, we introduce three diagnostic interventions, namely (i) instance-wise symbol anonymization, (ii) compact plan serialization, and (iii) verifier-reward fine-tuning using the VAL validator as a success-focused reinforcement signal. Symbol anonymization and compact serialization cause significant performance drops despite preserving plan semantics, thus revealing strong sensitivity to surface representations. Verifier-reward fine-tuning reaches performance saturation in half the supervised training epochs, but does not improve cross-domain generalization. For the explored configurations, in-domain performance plateaus around 80%, while cross-domain performance collapses, suggesting that our fine-tuned model relies heavily on domain-specific patterns rather than transferable planning competence in this setting. Our results highlight a persistent generalization gap in LLM-based planning and provide diagnostic tools for studying its causes.

On the Generalization Gap in LLM Planning: Tests and Verifier-Reward RL

TL;DR

This work investigates whether fine-tuned LLMs acquire transferable planning competence across PDDL-compatible domains or merely domain-specific patterns. It fine-tunes a 1.7B model on 40,000 domain–problem–plan tuples from IPC 2023 and evaluates in-domain and cross-domain generalization, revealing a strong in-domain performance (82.9% valid plans) but a complete generalization failure on unseen domains. Through three diagnostic variants—symbol anonymization, compact plan encoding, and verifier-reward RL—the study shows reliance on surface representations and domain-specific regularities, with limited gains in transfer despite improved training efficiency using VAL-based rewards. The work provides a multi-domain benchmark, diagnostic tools, and a VAL-based RL framework to probe and improve generalization in LLM-based planning, highlighting a persistent generalization gap and guiding future directions toward more abstract planning competence.

Abstract

Recent work shows that fine-tuned Large Language Models (LLMs) can achieve high valid plan rates on PDDL planning tasks. However, it remains unclear whether this reflects transferable planning competence or domain-specific memorization. In this work, we fine-tune a 1.7B-parameter LLM on 40,000 domain-problem-plan tuples from 10 IPC 2023 domains, and evaluate both in-domain and cross-domain generalization. While the model reaches 82.9% valid plan rate in in-domain conditions, it achieves 0% on two unseen domains. To analyze this failure, we introduce three diagnostic interventions, namely (i) instance-wise symbol anonymization, (ii) compact plan serialization, and (iii) verifier-reward fine-tuning using the VAL validator as a success-focused reinforcement signal. Symbol anonymization and compact serialization cause significant performance drops despite preserving plan semantics, thus revealing strong sensitivity to surface representations. Verifier-reward fine-tuning reaches performance saturation in half the supervised training epochs, but does not improve cross-domain generalization. For the explored configurations, in-domain performance plateaus around 80%, while cross-domain performance collapses, suggesting that our fine-tuned model relies heavily on domain-specific patterns rather than transferable planning competence in this setting. Our results highlight a persistent generalization gap in LLM-based planning and provide diagnostic tools for studying its causes.
Paper Structure (22 sections, 10 equations, 4 figures, 3 tables)

This paper contains 22 sections, 10 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Generation
  4. Models and Training Setups
  5. v1: Instance-wise Symbol Anonymization
  6. v2: Compact Plan Encoding
  7. v3: Verifier-Reward Fine-Tuning
  8. Experimental Protocol
  9. Results
  10. Baseline In-Domain Performance
  11. Unseen-Domain Generalization
  12. Diagnostic Variant Results
  13. Data Generation with Verifier-Reward
  14. Sensitivity to Representation
  15. Limitations
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Overview of the Gideon-based data generation pipeline and domain distribution.
  • Figure 2: The three SFT pipelines. In all three cases, the model receives domain and problem as input, and the generated plan is compared to the reference plan produced by Probe. The diagram on the top shows how we get the baseline B. The diagram in the mid illustrates the anonymization step for variant v1. The diagram on the bottom shows the compact encoding stage that simplifies the syntax of the reference PDDL plans for variant v2.
  • Figure 3: The RL pipeline used in this work. The model takes a domain–problem pair and, via GRPO rollouts, generates multiple candidate plans, which are decoded (timestamps and brackets restored), validated by VAL, and scored.
  • Figure 4: Variant v3 reaches approximately 80% validity in 1.5 total epochs (1 SFT + 0.5 RL). This exceeds the 2-epoch supervised baseline and approaches the 3-epoch ceiling.