Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering

Jim Maar, Denis Paperno, Callum Stuart McDougall, Neel Nanda

TL;DR

The paper tackles whether large language models implicitly plan during generation by introducing a simple, scalable activation-steering framework. It applies forward and backward planning probes to rhymed poetry and question answering, using mean activation differences to estimate steering vectors and manipulate planning signals. Across 23 diverse models (1B–32B, base and instruction-tuned), the study provides quantitative metrics for rhyming and QA planning, and demonstrates that steering can bias intermediate representations to influence both the ending rhyme and the generated answer. The results reveal that implicit planning is a pervasive mechanism that emerges even in smaller models, with planning strength scaling with size and instruction tuning, and suggest important considerations for AI safety and controllability.

Abstract

Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation to a predicted future token, such as a likely rhyming word, as supported by a prior qualitative study of Claude 3.5 Haiku using a cross-layer transcoder. We propose much simpler techniques for assessing implicit planning in language models. With case studies on rhyme poetry generation and question answering, we demonstrate that our methodology easily scales to many models. Across models, we find that the generated rhyme (e.g. "-ight") or answer to a question ("whale") can be manipulated by steering at the end of the preceding line with a vector, affecting the generation of intermediate tokens leading up to the rhyme or answer word. We show that implicit planning is a universal mechanism, present in smaller models than previously thought, starting from 1B parameters. Our methodology offers a widely applicable direct way to study implicit planning abilities of LLMs. More broadly, understanding planning abilities of language models can inform decisions in AI safety and control.

What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering

TL;DR

The paper tackles whether large language models implicitly plan during generation by introducing a simple, scalable activation-steering framework. It applies forward and backward planning probes to rhymed poetry and question answering, using mean activation differences to estimate steering vectors and manipulate planning signals. Across 23 diverse models (1B–32B, base and instruction-tuned), the study provides quantitative metrics for rhyming and QA planning, and demonstrates that steering can bias intermediate representations to influence both the ending rhyme and the generated answer. The results reveal that implicit planning is a pervasive mechanism that emerges even in smaller models, with planning strength scaling with size and instruction tuning, and suggest important considerations for AI safety and controllability.

Abstract

Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next token in preparation to a predicted future token, such as a likely rhyming word, as supported by a prior qualitative study of Claude 3.5 Haiku using a cross-layer transcoder. We propose much simpler techniques for assessing implicit planning in language models. With case studies on rhyme poetry generation and question answering, we demonstrate that our methodology easily scales to many models. Across models, we find that the generated rhyme (e.g. "-ight") or answer to a question ("whale") can be manipulated by steering at the end of the preceding line with a vector, affecting the generation of intermediate tokens leading up to the rhyme or answer word. We show that implicit planning is a universal mechanism, present in smaller models than previously thought, starting from 1B parameters. Our methodology offers a widely applicable direct way to study implicit planning abilities of LLMs. More broadly, understanding planning abilities of language models can inform decisions in AI safety and control.
Paper Structure (45 sections, 2 equations, 34 figures, 3 tables)

This paper contains 45 sections, 2 equations, 34 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. 3.1 Data
  5. 3.2 Models
  6. 3.3 Mean Activation Steering and Choosing a Steering Vector
  7. 3.4 Rhyming Metrics
  8. Fraction of Correct Rhyme Family.
  9. Fraction of Correct Rhyme Family (Steered).
  10. Fraction of Correct Last Word Regeneration.
  11. Fraction of Correct Last Word Regeneration (Steered).
  12. 3.5 Question answering metrics
  13. Fraction of Correct Answer
  14. Fraction of Correct Answer (Steered)
  15. Fraction of a/an
  16. ...and 30 more sections

Figures (34)

  • Figure 1: Comparison of unsteered regeneration rates to baseline chance level for the model. During regeneration with stochastic sampling, some rhyme families are expected to occur with non-zero frequency by chance. We estimate the baseline chance level that unsteered regeneration rate must exceed to show evidence for successful backward planning as the average frequency of rhyme families other than the one in whose context the couplet's second line was originally generated by the model.
  • Figure 2: Baseline rhyming abilities of models vs. steered rhyming behavior. Solid bars report the frequency of (exact) rhymes produced by the model (baseline rhyming behavior). Hashed bars present the frequency of (exact) rhymes of rhyme family 2 after the first line ending in a word from rhyme family 1, when steered towards rhyme family 2.
  • Figure 3: Baseline vs. steered last word regeneration rate of different models. Solid bars represent the fraction of times when regenerating the last word of the second line with the first line removed produces a word from the original rhyme family. This suggests that the model generates the second line so that it facilitates the correct rhyme, providing evidence for successful backward planning. Hashed bars represent the fraction of times when regenerating (with the first line removed) the last word of the second line generated with steering towards rhyme family 2 produces a word from rhyme family 2. Close match between baseline and steered regeneration rates suggests that we successfully manipulate the LLM's rhyme planning.
  • Figure 4: Regeneration rates per rhyme family with Gemma3 27B, baseline vs. steering. Solid bars represent the fraction of times when regenerating the last word of the second line with the first line removed produces a word from the original rhyme family. Hashed bars represent the fraction of times when regenerating (with the first line removed) the last word of the second line generated with steering towards the rhyme family produces a word from the target rhyme family. Close match between baseline and steered regeneration rates suggests that we successfully manipulate the LLM's rhyme planning for specific families.
  • Figure 5: Steering on last word (solid bars) vs. newline token (hashed bars) at the end of the first line of a couplet. Both positions produce comparable metrics for Gemma2 9B and Gemma3 27B. Steering effectiveness is the fraction of times rhyme family 2 is generated when steering towards rhyme family 2; steering on the newline token is slightly less effective.
  • ...and 29 more figures

Theorems & Definitions (7)

  • Definition 1.1
  • Definition 1.2
  • Definition 1.3
  • Definition 1.4
  • Example 1
  • Example 2
  • Example 3