Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FreeAct: Freeing Activations for LLM Quantization

Xiaohao Liu, Xiaobo Xia, Manyi Zhang, Ji-Fu Li, Xianzhi Yu, Fei Shen, Xiu Su, See-Kiong Ng, Tat-Seng Chua

TL;DR

This work proposes FreeAct, a novel quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities, and leverages the rank-deficient nature of activations to derive a solution space that extends beyond simple inverse matrices, enabling the decoupling of activation transformations from weights.

Abstract

Quantization is pivotal for mitigating the significant memory and computational overhead of Large Language Models (LLMs). While emerging transformation-based methods have successfully enhanced quantization by projecting feature spaces onto smoother manifolds using orthogonal matrices, they typically enforce a rigid one-to-one transformation constraint. This static approach fails to account for the dynamic patterns inherent in input activations, particularly within diffusion LLMs (dLLMs) and Multimodal LLMs (MLLMs), where varying token types exhibit distinct distributions. To advance this, we propose FreeAct, a novel quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities. Theoretically, we leverage the rank-deficient nature of activations to derive a solution space that extends beyond simple inverse matrices, enabling the decoupling of activation transformations from weights. Methodologically, FreeAct identifies token-specific dynamics (i.e., vision v.s. text, or masked tokens) and allocates distinct transformation matrices to the activation side, while maintaining a unified, static transformation for the weights. Extensive experiments across dLLMs and MLLMs demonstrate that FreeAct significantly outperforms baselines, up to 5.3% performance improvement, with in-depth analyses. Our code will be publicly released.

FreeAct: Freeing Activations for LLM Quantization

TL;DR

This work proposes FreeAct, a novel quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities, and leverages the rank-deficient nature of activations to derive a solution space that extends beyond simple inverse matrices, enabling the decoupling of activation transformations from weights.

Abstract

Quantization is pivotal for mitigating the significant memory and computational overhead of Large Language Models (LLMs). While emerging transformation-based methods have successfully enhanced quantization by projecting feature spaces onto smoother manifolds using orthogonal matrices, they typically enforce a rigid one-to-one transformation constraint. This static approach fails to account for the dynamic patterns inherent in input activations, particularly within diffusion LLMs (dLLMs) and Multimodal LLMs (MLLMs), where varying token types exhibit distinct distributions. To advance this, we propose FreeAct, a novel quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities. Theoretically, we leverage the rank-deficient nature of activations to derive a solution space that extends beyond simple inverse matrices, enabling the decoupling of activation transformations from weights. Methodologically, FreeAct identifies token-specific dynamics (i.e., vision v.s. text, or masked tokens) and allocates distinct transformation matrices to the activation side, while maintaining a unified, static transformation for the weights. Extensive experiments across dLLMs and MLLMs demonstrate that FreeAct significantly outperforms baselines, up to 5.3% performance improvement, with in-depth analyses. Our code will be publicly released.
Paper Structure (34 sections, 2 theorems, 22 equations, 16 figures, 2 tables, 1 algorithm)

This paper contains 34 sections, 2 theorems, 22 equations, 16 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Quantization
  4. Multimodal and Diffusion LLMs
  5. Methodology
  6. Beyond One-to-One
  7. FreeAct
  8. Guarantees and Implementations
  9. Experiments
  10. Experimental Setup
  11. Overall Performance (RQ1)
  12. Ablation Study (RQ2)
  13. Further Analysis (RQ3)
  14. Discussions and Outlook
  15. Theoretical Analysis
  16. ...and 19 more sections

Key Result

Proposition 1

If satisfy the equation of ${\bm{X}}{\bm{P}}\widetilde{{\bm{P}}}{\bm{W}}^{\top}= {\bm{X}}{\bm{W}}^{\top}$, the product ${\bm{P}}\widetilde{{\bm{P}}}$ belongs to the set which strictly contains the singleton $\{\mathbf{I}\}$. Consequently, the constraint admits a solution space that is strictly larger than the set of exact inverses (for which ${\bm{P}}\widetilde{{\bm{P}}}= {\bm{I}}$).

Figures (16)

  • Figure 1: The steep activation is smoothed by the designed transformation matrix for quantization (top). One weight corresponds to one transformation to ensure equivalence, i.e., ${\bm{P}}\times{\bm{P}}^{-1}= {\bm{I}}$. In practice, activations are diverse, reflecting dynamic patterns, necessitating flexible transformation matrices on the activation side, while keeping static on the weight side for quantization (bottom).
  • Figure 4: The overall framework of FreeAct. Different tokens are indexed to different transformation matrices according to their unique types (vision-text for MLLMs, and unmasked and masked for dLLMs). Different transformation matrices maintain a shared part together while possessing their own uniqueness, where zeros are filled in on another portion. The weight transformation matrix unites both portions to handle all the different activations. And the quantization error is minimized to optimize the quantization parameter.
  • Figure 5: Performance comparison under different rank-deficient settings, tested on LLaDA (left) and Qwen2.5VL (right).
  • Figure 6: Quantization errors optimized along with the training steps across different layers. ${\bm{X}}$ and ${\bm{X}}'$ denote different activation types, i.e., masked and unmasked token. The values in the left figure are normalized per layer.
  • Figure 7: Activation distributions after the transformation.
  • ...and 11 more figures

Theorems & Definitions (5)

  • Proposition 1: Beyond the Inverse
  • Theorem 2: Equivalence under projection invariance
  • proof
  • proof
  • proof