Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Can Take False First Steps at Inference-time Planning

Haijiang Yan, Jian-Qiao Zhu, Adam Sanborn

TL;DR

A Bayesian account for this gap is proposed by grounding planning behavior in the evolving generative context: given the subtle differences between natural language and the language internalized by LLMs, accumulated self-generated context drives a planning-shift during inference and thereby creates the appearance of compromised planning behavior.

Abstract

Large language models (LLMs) have been shown to acquire sequence-level planning abilities during training, yet their planning behavior exhibited at inference time often appears short-sighted and inconsistent with these capabilities. We propose a Bayesian account for this gap by grounding planning behavior in the evolving generative context: given the subtle differences between natural language and the language internalized by LLMs, accumulated self-generated context drives a planning-shift during inference and thereby creates the appearance of compromised planning behavior. We further validate the proposed model through two controlled experiments: a random-generation task demonstrating constrained planning under human prompts and increasing planning strength as self-generated context accumulates, and a Gaussian-sampling task showing reduced initial bias when conditioning on self-generated sequences. These findings provide a theoretical explanation along with empirical evidence for characterizing how LLMs plan ahead during inference.

Large Language Models Can Take False First Steps at Inference-time Planning

TL;DR

A Bayesian account for this gap is proposed by grounding planning behavior in the evolving generative context: given the subtle differences between natural language and the language internalized by LLMs, accumulated self-generated context drives a planning-shift during inference and thereby creates the appearance of compromised planning behavior.

Abstract

Large language models (LLMs) have been shown to acquire sequence-level planning abilities during training, yet their planning behavior exhibited at inference time often appears short-sighted and inconsistent with these capabilities. We propose a Bayesian account for this gap by grounding planning behavior in the evolving generative context: given the subtle differences between natural language and the language internalized by LLMs, accumulated self-generated context drives a planning-shift during inference and thereby creates the appearance of compromised planning behavior. We further validate the proposed model through two controlled experiments: a random-generation task demonstrating constrained planning under human prompts and increasing planning strength as self-generated context accumulates, and a Gaussian-sampling task showing reduced initial bias when conditioning on self-generated sequences. These findings provide a theoretical explanation along with empirical evidence for characterizing how LLMs plan ahead during inference.
Paper Structure (14 sections, 7 equations, 4 figures, 2 tables)

This paper contains 14 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. A Bayesian Account of Sequence-level Planning
  3. Meta-learned Bayesian Prior
  4. Compromised Planning during Inference
  5. Predicted Dynamic Signatures of Planning
  6. Experiment 1: Planning Shift and Convergence in Random Generation Task
  7. Data Analysis
  8. Results
  9. Discussion
  10. Experiment 2: Biasing-then-debiasing in Gaussian Sampling Task
  11. Data Analysis
  12. Results
  13. Discussion
  14. General Discussion

Figures (4)

  • Figure 1: Scheme illustrating the evolving planning behavior of LLMs during inference.
  • Figure 2: Percentage of variance in future tokens ($R^2$ of regressions; vertical axis) captured by the LLM embeddings across different planning horizons (offset $\Delta t$; horizontal axis): (top) Llama-3.1-8B-Instruct, (bottom) Qwen-2.5-7B-Instruct.
  • Figure 3: Relationships between plan adherence and planning position: (left) Llama-3.1-8B-Instruct, layer 15-25 from lightest to darkest, (right) Qwen-2.5-7B-Instruct, layer 10-20 from lightest to darkest. Higher $R^2$ indicates stronger adherence of later token generations to the latent plans formed eight steps earlier.
  • Figure 4: Generation start-off dynamic patterns in Gaussian sampling task: (left) Llama-3.1-8B-Instruct, (right) Qwen-2.5-7B-Instruct. Both panels display the mean with 95% confidence intervals of samples at each generation position, with the seven conditions presented vertically. Within each condition, three sets of samples are shown from left to right: (i) samples from a Gaussian distribution $\mathcal{N}(\mu, 10)$, (ii) samples generated by the LLM conditioned on the Gaussian (Gen. I), and (iii) samples generated by the LLM conditioned on Gen. I (Gen. II). We find statistically significant negative biases in the initial samples of Gen. I (see Tables \ref{['ttest_starting_bias_llama']} and \ref{['ttest_starting_bias_qwen']} for details).