On the Emergence and Test-Time Use of Structural Information in Large Language Models

TL;DR

The paper tackles whether large language models learn and can use structural information to perform test-time compositional generation. It introduces a Transformational Grammar–based synthetic dataset to study the emergence of structural knowledge and how it supports (or fails to support) compositional generation under fine-tuning and LoRA regimes. Key findings include a phase-transition in internal representations around $64k$ training steps, where $d(s,A(s))$ grows and aligns with gains in reasoning tasks, and ablations indicating MLPs as primary drivers of syntactic transformations; however, test-time generalization to unseen nested transformations remains limited. This work advances mechanistic interpretability by showing how structured knowledge forms and is used (or constrained) in LLMs, and provides a controlled benchmark for future studies on robust, compositional language capabilities.

Abstract

Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex reasoning tasks, and that the ability to perform test-time compositional generation remains limited.

Figures (9)

• Figure 1: Example of a model prompt for the NP-raising transformation task (top) and a sample output generated by the model (bottom).
• Figure 2: L2 transformation norms and Pythia-410M performance training. Red lines show syntactic transformation embedding differences. Blue lines show model performance: (Top) downstream task accuracy on reasoning tasks. (Bottom) language modeling perplexity on WikiText and Paloma datasets. Individual metrics shown as dashed lines, averages as solid lines.
• Figure 3: Prompt formats used during fine-tuning and inference.
• Figure 4: (Top) Relative contribution of MLP compared to multi-head attention (Bottom) Progression of probability of predicting the correct token for each transformation over all layers
• Figure 5: Heatmap showing average projections of intermediate representations onto the transformation A vs. others direction across all 32 layers. Each row represents a different transformation type (A-J), with transformation A contrasted against all other transformations. Red regions indicate higher projection values (representations more similar to transformation A), while blue regions indicate lower projection values (representations more dissimilar to transformation A).

Theorems & Definitions (3)

• Definition
• Proposition
• proof : Proof of Proposition \ref{['prop:twocases']}