From Compression to Expression: A Layerwise Analysis of In-Context Learning
Jiachen Jiang, Yuxin Dong, Jinxin Zhou, Zhihui Zhu
TL;DR
The paper investigates how in-context learning encodes task information across layers in large language models, uncovering a Layerwise Compression-Expression dynamic where early layers compress demonstration-derived task signals and later layers express them to generate outputs. It introduces the TDNV metric to quantify within-task compression versus between-task separation and demonstrates this phenomenon universally across architectures and tasks, with larger models and more demonstrations yielding stronger compression and better ICL performance. A bias-variance decomposition of task vectors shows that both bias and variance decay on the order of $\mathcal{O}(1/K)$ as the number of demonstrations $K$ increases, and analyses reveal how attention contributes to this reduction. The work also provides practical insights, such as using minimum-TDNV to identify optimal task-vector layers and applying contrastive fine-tuning to further compress task representations, which improves task-vector accuracy by about 20% on symbolic tasks. Overall, the findings offer a principled view of internal representation dynamics in ICL, with implications for interpretability, robustness to noisy demonstrations, and efficiency in large-scale models.
Abstract
In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks without weight updates by learning from demonstration sequences. While ICL shows strong empirical performance, its internal representational mechanisms are not yet well understood. In this work, we conduct a statistical geometric analysis of ICL representations to investigate how task-specific information is captured across layers. Our analysis reveals an intriguing phenomenon, which we term *Layerwise Compression-Expression*: early layers progressively produce compact and discriminative representations that encode task information from the input demonstrations, while later layers express these representations to incorporate the query and generate the prediction. This phenomenon is observed consistently across diverse tasks and a range of contemporary LLM architectures. We demonstrate that it has important implications for ICL performance -- improving with model size and the number of demonstrations -- and for robustness in the presence of noisy examples. To further understand the effect of the compact task representation, we propose a bias-variance decomposition and provide a theoretical analysis showing how attention mechanisms contribute to reducing both variance and bias, thereby enhancing performance as the number of demonstrations increases. Our findings reveal an intriguing layerwise dynamic in ICL, highlight how structured representations emerge within LLMs, and showcase that analyzing internal representations can facilitate a deeper understanding of model behavior.