Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

Amit LeVi, Raz Lapid, Rom Himelstein, Yaniv Nemcovsky, Ravid Shwartz Ziv, Avi Mendelson

TL;DR

The paper addresses the inefficiency of deploying large LLMs when only a subset of tasks is needed by introducing task-aware PTQ methods. TAQ assigns per-layer bitwidths based on task-conditioned activation statistics, while TAQO uses direct task-sensitivity measurements to protect critical layers under a budget, both aiming to preserve end-task performance with substantial compression. The authors quantify layer relevance using information and stability scores derived from activations, and they validate the approach across open-weight LLMs on QA, math, and code tasks, showing near full-precision accuracy with significant reductions in size and latency. This work highlights that task signals are unevenly distributed across depth, enabling both practical deployment gains and deeper insights into how tasks are encoded within Transformer layers.

Abstract

Large Language Models (LLMs) excel across diverse tasks, yet many applications require only limited capabilities, making large variants inefficient in memory and latency. Existing approaches often combine distillation and quantization, but most post-training quantization (PTQ) methods are task-agnostic, ignoring how task-specific signals are distributed across layers. In this work, we propose to use hidden representations that encode task-salient signals as a guideline for quantization. In order to fully utilize our innovative idea, this paper compares two new task-aware PTQ methods: Task-Aware Quantization (TAQ), which allocates bitwidths using task-conditioned statistics from hidden activations, and TAQO, which allocates precision based on direct layer sensitivity tests. From a small calibration set, these approaches identify task-relevant layers, preserving their precision while aggressively quantizing the rest. This yields stable task sensitivity profiles and efficient task-specialized models. Across models, TAQ and TAQO outperform the baselines; TAQ leads on Phi-4, while TAQO leads on Llama-3.1, Qwen3, and Qwen2.5. For instances, on Phi-4 it achieves 42.33 EM / 50.81 F1, far surpassing Activation-aware Weight Quantization (AWQ) (2.25 / 7.07), while remaining within < 1.0% of the original accuracy at lower average precision.

You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

TL;DR

The paper addresses the inefficiency of deploying large LLMs when only a subset of tasks is needed by introducing task-aware PTQ methods. TAQ assigns per-layer bitwidths based on task-conditioned activation statistics, while TAQO uses direct task-sensitivity measurements to protect critical layers under a budget, both aiming to preserve end-task performance with substantial compression. The authors quantify layer relevance using information and stability scores derived from activations, and they validate the approach across open-weight LLMs on QA, math, and code tasks, showing near full-precision accuracy with significant reductions in size and latency. This work highlights that task signals are unevenly distributed across depth, enabling both practical deployment gains and deeper insights into how tasks are encoded within Transformer layers.

Abstract

Large Language Models (LLMs) excel across diverse tasks, yet many applications require only limited capabilities, making large variants inefficient in memory and latency. Existing approaches often combine distillation and quantization, but most post-training quantization (PTQ) methods are task-agnostic, ignoring how task-specific signals are distributed across layers. In this work, we propose to use hidden representations that encode task-salient signals as a guideline for quantization. In order to fully utilize our innovative idea, this paper compares two new task-aware PTQ methods: Task-Aware Quantization (TAQ), which allocates bitwidths using task-conditioned statistics from hidden activations, and TAQO, which allocates precision based on direct layer sensitivity tests. From a small calibration set, these approaches identify task-relevant layers, preserving their precision while aggressively quantizing the rest. This yields stable task sensitivity profiles and efficient task-specialized models. Across models, TAQ and TAQO outperform the baselines; TAQ leads on Phi-4, while TAQO leads on Llama-3.1, Qwen3, and Qwen2.5. For instances, on Phi-4 it achieves 42.33 EM / 50.81 F1, far surpassing Activation-aware Weight Quantization (AWQ) (2.25 / 7.07), while remaining within < 1.0% of the original accuracy at lower average precision.

Paper Structure

This paper contains 25 sections, 13 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Quantization: from uniform to layer- and activation-aware.
  4. Per-layer activation analysis and directionality.
  5. Layer hidden representations.
  6. Task signals within LLM activation space.
  7. External representation alignment and invariances.
  8. Connecting PTQ with representation analyses.
  9. Method
  10. Motivation.
  11. Objective.
  12. Task-Aware Quantization (TAQ)
  13. Scoring: information and stability.
  14. Calibration: quantizer tuning.
  15. Task-Aware Quantization with Oracle Layer Selection (TAQO)
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Layers relevance scores per task.