Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Model

Navin Ranjan, Andreas Savakis

TL;DR

Mix-QSAM, a mixed-precision PTQ framework for SAM, is proposed, which introduces a layer-wise importance score, derived using Kullback–Leibler (KL) divergence, and introduces cross-layer synergy, a novel metric based on causal mutual information, to capture dependencies between adjacent layers.

Abstract

The Segment Anything Model (SAM) is a popular vision foundation model; however, its high computational and memory demands make deployment on resource-constrained devices challenging. While Post-Training Quantization (PTQ) is a practical approach for reducing computational overhead, existing PTQ methods rely on fixed bit-width quantization, leading to suboptimal accuracy and efficiency. To address this limitation, we propose Mix-QSAM, a mixed-precision PTQ framework for SAM. First, we introduce a layer-wise importance score, derived using Kullback-Leibler (KL) divergence, to quantify each layer's contribution to the model's output. Second, we introduce cross-layer synergy, a novel metric based on causal mutual information, to capture dependencies between adjacent layers. This ensures that highly interdependent layers maintain similar bit-widths, preventing abrupt precision mismatches that degrade feature propagation and numerical stability. Using these metrics, we formulate an Integer Quadratic Programming (IQP) problem to determine optimal bit-width allocation under model size and bit-operation constraints, assigning higher precision to critical layers while minimizing bit-width in less influential layers. Experimental results demonstrate that Mix-QSAM consistently outperforms existing PTQ methods on instance segmentation and object detection tasks, achieving up to 20% higher average precision under 6-bit and 4-bit mixed-precision settings, while maintaining computational efficiency.

Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Model

TL;DR

Mix-QSAM, a mixed-precision PTQ framework for SAM, is proposed, which introduces a layer-wise importance score, derived using Kullback–Leibler (KL) divergence, and introduces cross-layer synergy, a novel metric based on causal mutual information, to capture dependencies between adjacent layers.

Abstract

The Segment Anything Model (SAM) is a popular vision foundation model; however, its high computational and memory demands make deployment on resource-constrained devices challenging. While Post-Training Quantization (PTQ) is a practical approach for reducing computational overhead, existing PTQ methods rely on fixed bit-width quantization, leading to suboptimal accuracy and efficiency. To address this limitation, we propose Mix-QSAM, a mixed-precision PTQ framework for SAM. First, we introduce a layer-wise importance score, derived using Kullback-Leibler (KL) divergence, to quantify each layer's contribution to the model's output. Second, we introduce cross-layer synergy, a novel metric based on causal mutual information, to capture dependencies between adjacent layers. This ensures that highly interdependent layers maintain similar bit-widths, preventing abrupt precision mismatches that degrade feature propagation and numerical stability. Using these metrics, we formulate an Integer Quadratic Programming (IQP) problem to determine optimal bit-width allocation under model size and bit-operation constraints, assigning higher precision to critical layers while minimizing bit-width in less influential layers. Experimental results demonstrate that Mix-QSAM consistently outperforms existing PTQ methods on instance segmentation and object detection tasks, achieving up to 20% higher average precision under 6-bit and 4-bit mixed-precision settings, while maintaining computational efficiency.
Paper Structure (16 sections, 13 equations, 2 figures, 5 tables)

This paper contains 16 sections, 13 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Segment Anything Model
  4. Vision Transformer Quantization
  5. Method
  6. Model Quantization
  7. Causal Layer Importance
  8. Cross-Layer Synergy
  9. Mixed-Precision Bit Allocation
  10. Experiments
  11. Results on Instance Segmentation
  12. Results on Object Detection
  13. Qualitative Results
  14. Ablation Study
  15. Efficiency Analysis
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Mixed-precision bit allocation for SAM- under W6/A6: comparing layer importance score alone vs. using both layer importance score and cross-layer synergy. Candidate bit widths are $\{4, 5, 6, 7, 8\}$
  • Figure 2: Qualitative comparison of instance segmentation on the COCO dataset using a 4-bit Segment Anything Model (SAM) with YOLOX bounding boxes as prompts. (a) Original images; (b) Full-precision SAM; (c) 4-bit PTQ4SAM-S without reconstruction; (d) 4-bit PTQ4SAM-L with reconstruction; (e) 4-bit Mix-QSAM (ours) without reconstruction.