Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

TL;DR

This work shows that MAW-guided width pruning of GLU-MLP layers in Llama-3.2 does not merely compress models but selectively reshapes capabilities. It reveals a consistent Capabilities Dichotomy where parametric-knowledge tasks degrade while instruction-following and truthfulness-related behaviors are preserved or enhanced, linked to an inverse MMLU–TruthfulQA relationship (r = -0.864, p = 0.012). The study identifies a practical equilibrium at an expansion ratio of $2.4\times$ that balances performance and energy efficiency, with batch processing offering substantial throughput gains and single-shot latency costs. These findings have implications for deploying pruned models in resource-constrained settings and motivate further exploration of architecture-aware pruning as a tool for selective behavioral modification across GLU-based transformers.

Abstract

Structured width pruning of GLU-MLP layers, guided by the Maximum Absolute Weight (MAW) criterion, reveals a systematic dichotomy in how reducing the expansion ratio affects different model capabilities. While performance on tasks relying on parametric knowledge (e.g., MMLU, GSM8K) and perplexity metrics degrades predictably, instruction-following capabilities improve substantially (+46% to +75% in IFEval for Llama-3.2-1B and 3B models), and multi-step reasoning remains robust (MUSR). This pattern challenges the prevailing assumption that pruning induces uniform degradation. We evaluated seven expansion ratio configurations using comprehensive benchmarks assessing factual knowledge, mathematical reasoning, language comprehension, instruction-following, and truthfulness. Our analysis identifies the expansion ratio as a critical architectural parameter that selectively modulates cognitive capabilities, rather than merely serving as a compression metric. We provide the first systematic characterization of this selective preservation phenomenon. Notably, we document a robust inverse correlation (r = -0.864, p = 0.012 in Llama-3B) between factual knowledge capacity (MMLU) and truthfulness metrics (TruthfulQA-MC2): as knowledge degrades, the model's ability to discriminate misconceptions improves consistently. This connects two previously distinct research areas, demonstrating that MAW-guided width pruning acts as a selective filter, reducing parametric knowledge while preserving or enhancing behavioral alignment. Additionally, we quantify context-dependent efficiency trade-offs: pruned configurations achieve up to 23% reduction in energy consumption (J/token) but incur penalties in single-request latency, whereas batch processing workloads benefit uniformly.

Figures (5)

• Figure 1: GLU (Gated Linear Unit) architecture within the MLP block of Llama-3.2-1B. The diagram shows the input flow from post_attention_layernom, the parallel up_proj and gate_proj projections, the SiLU activation function (act_fn), the element-wise multiplication (gating), and the final down_proj contraction.
• Figure 2: Llama-3.2-1B Benchmarks. Panel A (Fragile Capabilities) shows the predictable collapse of knowledge-dependent tasks (GSM8K, Lambada, WikiText) as expansion ratio decreases. Panel B (Robust Capabilities) reveals the contrasting improvement of algorithmic and instruction-following tasks (IFEval, MUSR, TruthfulQA-MC2). Performance is normalized to baseline (4.0√ó) = 100%. The X-axis represents expansion ratios from 4.0√ó to 1.6√ó; the Y-axis shows normalized performance.
• Figure 3: Llama-3.2-3B Benchmarks. Panel A (Fragile Capabilities) demonstrates monotonic degradation of knowledge-dependent tasks (GSM8K, Lambada, WikiText) across expansion ratios 2.67√ó to 1.07√ó. Panel B (Robust Capabilities) exhibits non-monotonic improvement in instruction-following (IFEval reaching 174.4% of baseline) and consistent gains in truthfulness (TruthfulQA-MC2). Performance is normalized to baseline (2.67√ó) = 100%. This pattern replicates the dichotomy observed in Llama-1B.
• Figure 4: The Truthfulness Paradox. Divergent trajectories of factual knowledge (MMLU, blue dashed lines) and truthfulness (TruthfulQA-MC2, orange solid lines) across expansion ratios. Panel A shows Llama-3.2-1B; Panel B shows Llama-3.2-3B. As the expansion ratio decreases, factual knowledge degrades while truthfulness improves, revealing a systematic inverse relationship.
• Figure 5: Efficiency Trade-offs --- Single-Request vs Batch Processing. Panel A (Llama-3.2-1B) and Panel B (Llama-3.2-3B) display the relationship between end-to-end latency (red lines), single-request energy consumption (blue dashed lines), and batch processing energy consumption (blue solid lines) across expansion ratios. Latency increases with pruning intensity in both models. Single-request energy decreases across expansion ratios, while batch processing energy remains consistently lower than single-request configurations at all expansion ratios. The left Y-axis shows latency in milliseconds; the right Y-axis shows energy consumption in joules per token.