Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PTQ4VM: Post-Training Quantization for Visual Mamba

Younghyun Cho, Changhun Lee, Seonggon Kim, Eunhyeok Park

TL;DR

PTQ4VM analyzes quantization in Visual Mamba backbones, identifies token-wise variance, channel-wise outliers, and long-tail activations as core obstacles, and introduces Per-Token Static (PTS) quantization combined with Joint Learning of Smoothing Scale and Step Size (JLSS). By integrating with SmoothQuant for outliers and applying static truncation strategies, it achieves quantized models within 15 minutes and up to 1.83x GPU speedups with negligible FP16 accuracy loss across image classification, object detection, and segmentation tasks. The approach demonstrates broad applicability across Vision Mamba variants and emphasizes efficient, low-overhead PTQ for vision transformers with fixed token processing orders.

Abstract

Visual Mamba is an approach that extends the selective space state model, Mamba, to vision tasks. It processes image tokens sequentially in a fixed order, accumulating information to generate outputs. Despite its growing popularity for delivering high-quality outputs at a low computational cost across various tasks, Visual Mamba is highly susceptible to quantization, which makes further performance improvements challenging. Our analysis reveals that the fixed token access order in Visual Mamba introduces unique quantization challenges, which we categorize into three main issues: 1) token-wise variance, 2) channel-wise outliers, and 3) a long tail of activations. To address these challenges, we propose Post-Training Quantization for Visual Mamba (PTQ4VM), which introduces two key strategies: Per-Token Static (PTS) quantization and Joint Learning of Smoothing Scale and Step Size (JLSS). To the our best knowledge, this is the first quantization study on Visual Mamba. PTQ4VM can be applied to various Visual Mamba backbones, converting the pretrained model to a quantized format in under 15 minutes without notable quality degradation. Extensive experiments on large-scale classification and regression tasks demonstrate its effectiveness, achieving up to 1.83x speedup on GPUs with negligible accuracy loss compared to FP16. Our code is available at https://github.com/YoungHyun197/ptq4vm.

PTQ4VM: Post-Training Quantization for Visual Mamba

TL;DR

PTQ4VM analyzes quantization in Visual Mamba backbones, identifies token-wise variance, channel-wise outliers, and long-tail activations as core obstacles, and introduces Per-Token Static (PTS) quantization combined with Joint Learning of Smoothing Scale and Step Size (JLSS). By integrating with SmoothQuant for outliers and applying static truncation strategies, it achieves quantized models within 15 minutes and up to 1.83x GPU speedups with negligible FP16 accuracy loss across image classification, object detection, and segmentation tasks. The approach demonstrates broad applicability across Vision Mamba variants and emphasizes efficient, low-overhead PTQ for vision transformers with fixed token processing orders.

Abstract

Visual Mamba is an approach that extends the selective space state model, Mamba, to vision tasks. It processes image tokens sequentially in a fixed order, accumulating information to generate outputs. Despite its growing popularity for delivering high-quality outputs at a low computational cost across various tasks, Visual Mamba is highly susceptible to quantization, which makes further performance improvements challenging. Our analysis reveals that the fixed token access order in Visual Mamba introduces unique quantization challenges, which we categorize into three main issues: 1) token-wise variance, 2) channel-wise outliers, and 3) a long tail of activations. To address these challenges, we propose Post-Training Quantization for Visual Mamba (PTQ4VM), which introduces two key strategies: Per-Token Static (PTS) quantization and Joint Learning of Smoothing Scale and Step Size (JLSS). To the our best knowledge, this is the first quantization study on Visual Mamba. PTQ4VM can be applied to various Visual Mamba backbones, converting the pretrained model to a quantized format in under 15 minutes without notable quality degradation. Extensive experiments on large-scale classification and regression tasks demonstrate its effectiveness, achieving up to 1.83x speedup on GPUs with negligible accuracy loss compared to FP16. Our code is available at https://github.com/YoungHyun197/ptq4vm.
Paper Structure (27 sections, 9 equations, 10 figures, 7 tables)

This paper contains 27 sections, 9 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Selective State Space model
  4. Visual Mamba Backbones
  5. Post-training Quantization
  6. SmoothQuant
  7. Analysis for Quantization Target
  8. Challenges of Linear Layer Quantization
  9. Observation 1: Token-wise Variance
  10. Observation 2: Channel-wise Outliers
  11. Observation 3: Long tail of Activation
  12. PTQ4VM
  13. Per-Token Static (PTS) Quantization
  14. Joint Learning of Smoothing Scale and Step Size (JLSS)
  15. Experiments
  16. ...and 12 more sections

Figures (10)

  • Figure 1: Distribution of the input activations of a 22nd out_proj layer in Vim-Ti. (a) Images from 3 categories, (b) their corresponding activation distributions, and (c) the average values across (i) the channel dimension and (ii) the token dimension.
  • Figure 2: The Visual Mamba backbones consist of (a) Vision Mamba, (b) LocalVim, (c) VMamba, and (d) LocalVMamba. "x2" and "x4" indicate the repetition of operations based on scan directions. The square matrices beneath illustrate the scan method for each backbone.
  • Figure 3: Profiling results of Visual Mamba backbones on an RTX 3090. The numbers above the bars indicate the speedup.
  • Figure 4: Observation on activation distributions across (a) model size, (b) types of linear layers, and (c) block indices. (c) shows the activation distribution of the out_proj layer.
  • Figure 5: Comparison of activation distribution on LocalMamba backbone depending on whether CLS Token is utilized or not.
  • ...and 5 more figures