What Matters to an LLM? Behavioral and Computational Evidences from Summarization

TL;DR

This work addresses what information LLMs prioritize when generating summaries and how that latent notion is encoded. It introduces an empirical importance distribution $I_M(D)$ by generating $k=10$ length‑controlled summaries per document and analyzes both outputs and internal signals through attention and probing. The findings reveal that LLMs exhibit consistent, architecture‑driven importance patterns that cluster by model family, with middle‑to‑late transformer layers and certain attention heads aligning with $I_M(D)$, though the strength of alignment is dataset and language dependent. This provides a path toward interpretable and potentially controllable information selection in LLMs and supplies a substantial dataset of summaries to support further cross‑model analysis.

Abstract

Large Language Models (LLMs) are now state-of-the-art at summarization, yet the internal notion of importance that drives their information selections remains hidden. We propose to investigate this by combining behavioral and computational analyses. Behaviorally, we generate a series of length-controlled summaries for each document and derive empirical importance distributions based on how often each information unit is selected. These reveal that LLMs converge on consistent importance patterns, sharply different from pre-LLM baselines, and that LLMs cluster more by family than by size. Computationally, we identify that certain attention heads align well with empirical importance distributions, and that middle-to-late layers are strongly predictive of importance. Together, these results provide initial insights into what LLMs prioritize in summarization and how this priority is internally represented, opening a path toward interpreting and ultimately controlling information selection in these models.

Figures (22)

• Figure 1: Analytical framework for modeling information importance. (1) Behavioral Analysis: We generate length-variant summaries with LLMs across three datasets (CNN/DailyMail, SAMSum, DECODA-French). The importance distribution$I_M(D)$ is derived as summary persistence. (2) Attention Analysis: Raw attention weights are aggregated and normalized to obtain token-level distributions. (3) Probing: Hidden states are used to train probes in three scenarios (S1: Layer-wise/Token, S2: All-layers/Token, S3: Layer-wise/Article) to predict $I_M(D)$.
• Figure 2: Pairwise model similarity based on Spearman rank correlation distance for importance distributions, visualized via two-dimensional Multidimensional Scaling (MDS). Results are shown for the CNN/DailyMail (top) and SAMSum (bottom) datasets.
• Figure 3: Multi-Dimensional Scaling (MDS) projection of attention heads for Llama-3.2-1B-Instruct on SAMSum. Points represent heads positioned by their per-sample NDCG@10 profiles, reflecting similarity in top-$k$ ranking quality with importance distribution. Point color indicates layer depth (lighter = deeper). The red star marks the ideal point of perfect ranking alignment (NDCG@10 = 1.0); dashed contours indicate similarity thresholds (e.g., $\text{NDCG@10} \geq 0.9$).
• Figure 4: Article-level probing NDCG@10 across layers for CNN/DailyMail (top) and SAMSum (bottom). Round dots show learned model performance; square dots show the Randomized Weights Baseline. The best-performing layers are annotated. Horizontal dashed lines show the TextRank baseline for each model.
• Figure 5: Heatmap visualization of layer-wise importance predictions for Qwen2.5-3B-Instruct, showing probe outputs across all layers and the first 50 tokens for representative samples (top: CNN/DailyMail, bottom: SAMSum).