Void in Language Models

TL;DR

This work investigates whether all layers in transformer language models are activated during inference and introduces a non-training method, L2 Adaptive Computation (LAC), to detect Voids—unactivated layers—during both Prompt Processing and Response Generation. By adapting LAC to inference and applying a per-token halting framework with dynamic thresholds, the authors reveal phase-specific layer usage and demonstrate that skipping Voids can improve performance on several tasks using only a subset of layers. The study shows distinct PP and RG activation patterns across instruction-tuned LMs (Llama, Mistral, Qwen) and benchmarks (MMLU, GPQA Diamond, BoolQ), highlighting the non-uniform contribution of layers and offering a new lens for interpretability, targeted knowledge editing, and pruning strategies. While masking-based halting does not yet yield hardware speedups, the results point to significant opportunities for efficiency gains and deeper understanding of layer-specific roles in inference and reasoning tasks.

Abstract

Despite advances in transformer-based language models (LMs), a fundamental question remains largely unanswered: Are all layers activated during inference? We investigate this question by detecting unactivated layers (which we refer to as Voids) using a non-trainable and parameter-free adaptive computation method called L2 Adaptive Computation (LAC). We adapt LAC from its original efficiency-focused application to trace activated layers during inference. This method monitors changes in the L2-norm of activations to identify voids. We analyze layer activation in instruction-tuned LMs across two phases: Prompt Processing (PP), where we trace activated layers for each token in the input prompts, and Response Generation (RG), where we trace activated layers for each generated token. We further demonstrate that distinct layers are activated during these two phases. To show the effectiveness of our method, we evaluated three distinct instruction-tuned LMs from the Llama, Mistral, and Qwen families on three benchmarks: MMLU, GPQA Diamond, and BoolQ. For example, on MMLU with a zero-shot setting, skipping voids in Qwen2.5-7B-Instruct resulted in an improvement from 69.24 to 71.29 while the model uses only 30% of the layers. Similarly, Mistral-7B-Instruct-v0.3 on GPQA Diamond improved from 13.88 to 18.36 when using 70% of the layers during both the PP and RG phases. These results show that not all layers contribute equally during inference, and that selectively skipping most of them can improve the performance of models on certain tasks.

Figures (9)

• Figure 1: BoolQ accuracy and Percentage of total layers used (relative to total model layers) as a function of $\alpha$. Left: Llama-3-8B-Instruct; Middle: Mistral-7B-Instruct-v0.3; Right: Qwen2.5-7B-Instruct. In the top row, the blue line shows performance when skipping voids, and the red horizontal line indicates the full model's performance. In the bottom row, the red line represents average layer usage during Response Generation (RG), and the green line shows average layer usage during Prompt Processing (PP). Usage is shown as a percentage of total model layers.
• Figure 2: Usage percentage of layers across proposed LMs while evaluating them on stated benchmarks. Red lines represent the usage of each layer for the Prompt Processing phase, where blue lines show the usage percentage of layers for the Response Generation phase. For ease of demonstration, we normalized the usage for each layer.
• Figure 3: Mean L2 Norm (top row) and mean change in L2 Norm (bottom row) for each layer of Llama-3-8B-Instruct, Mistral-7-B-Instruct-v0.3, and Qwen2.5-7B-Instruct, averaged across the evaluated benchmarks. Blue lines represent metrics during the Prompt Processing (PP) phase, and red lines represent metrics during the Response Generation (RG) phase. The change in L2 norm for a layer is calculated from the L2 norm of its output minus the L2 norm of its input (see Section \ref{['ref:Halt']}).
• Figure :
• Figure :