How Do Language Models Understand Tables? A Mechanistic Analysis of Cell Location

TL;DR

Facing the opacity of how LLMs handle two-dimensional tabular data, the paper proposes a mechanistic account of table understanding framed as cell location. It identifies a three-stage information flow—Semantic Binding, Coordinate Localization, and Information Extraction—and demonstrates that coordinate resolution relies on delimiter counting and a linear subspace representation of column indices, steered via vector arithmetic. Across Qwen and Llama models and multiple serialization formats, the authors validate the causal roles of specific attention heads and show that multi-cell queries are solved by multiplexing the same atomic mechanisms in parallel. The work provides a principled, interpretable view of table reasoning with implications for improving reliability and robustness of tabular tasks in LLMs.

Abstract

While Large Language Models (LLMs) are increasingly deployed for table-related tasks, the internal mechanisms enabling them to process linearized two-dimensional structured tables remain opaque. In this work, we investigate the process of table understanding by dissecting the atomic task of cell location. Through activation patching and complementary interpretability techniques, we delineate the table understanding mechanism into a sequential three-stage pipeline: Semantic Binding, Coordinate Localization, and Information Extraction. We demonstrate that models locate the target cell via an ordinal mechanism that counts discrete delimiters to resolve coordinates. Furthermore, column indices are encoded within a linear subspace that allows for precise steering of model focus through vector arithmetic. Finally, we reveal that models generalize to multi-cell location tasks by multiplexing the identical attention heads identified during atomic location. Our findings provide a comprehensive explanation of table understanding within Transformer architectures.

Figures (23)

• Figure 1: The mechanism interpretation of the sequential three-stage pipeline for table understanding.
• Figure 2: Average column patching effect score across layers and token positions.
• Figure 3: Analysis of semantic binding mechanisms. (a) Top-1 accuracy evaluated with cosine similarity between query constraints and all table headers across layers. (b) The impact of ablating row alignment heads by observing the changes of outputs.
• Figure 4: Visualization of the geometric interaction score $S_{j,l}$ in coordinate-encoding heads. The heatmap illustrates the dot product between column-averaged query vectors ($\bar{q}_j$) and key vectors ($\bar{k}_l$) after injecting RoPE corresponding to their column indices.
• Figure 5: Effect Scores of patching attention heads at the last token position. The red boxes highlight heads that exhibit positive Effect Scores and attention the target cell significantly.