Unlocking the Future: Exploring Look-Ahead Planning Mechanistic Interpretability in Large Language Models

TL;DR

This work probes the mechanisms by which large language models perform planning in a fully observable Blocksworld task. It introduces a two-stage framework (information-flow analysis and internal-representation probing) and a Look-Ahead Planning Decisions Existence Hypothesis, demonstrating that middle-to-upper MHSA layers can decode planning decisions and that internal representations encode a subset of short-term future decisions, especially early in the network. The findings reveal that MHSA primarily draws on spans of goal states and recent steps, and that look-ahead information is stored in advance but becomes harder to maintain as planning depth increases. This study advances mechanistic interpretability of planning in LLMs and informs the design of planning-enabled agents and probing methods for future work.

Abstract

Planning, as the core module of agents, is crucial in various fields such as embodied agents, web navigation, and tool using. With the development of large language models (LLMs), some researchers treat large language models as intelligent agents to stimulate and evaluate their planning capabilities. However, the planning mechanism is still unclear. In this work, we focus on exploring the look-ahead planning mechanism in large language models from the perspectives of information flow and internal representations. First, we study how planning is done internally by analyzing the multi-layer perception (MLP) and multi-head self-attention (MHSA) components at the last token. We find that the output of MHSA in the middle layers at the last token can directly decode the decision to some extent. Based on this discovery, we further trace the source of MHSA by information flow, and we reveal that MHSA mainly extracts information from spans of the goal states and recent steps. According to information flow, we continue to study what information is encoded within it. Specifically, we explore whether future decisions have been encoded in advance in the representation of flow. We demonstrate that the middle and upper layers encode a few short-term future decisions to some extent when planning is successful. Overall, our research analyzes the look-ahead planning mechanisms of LLMs, facilitating future research on LLMs performing planning tasks.

Figures (16)

• Figure 1: An example of greedy and look-ahead planning.
• Figure 2: An example of Blocksworld.
• Figure 3: Extraction rate of different components in Llama-2-7b-chat-hf.
• Figure 4: Extraction rate of different components in Vicuna-7b.
• Figure 5: Information flow of last token in Llama-2-7b-chat-hf.