Fisher-aware Quantization for DETR Detectors with Critical-category Objectives

TL;DR

This work tackles the fine-grained impact of quantization on DETR detectors by defining a critical-category objective that focuses on a subset of important classes through a logit-label transformation. It establishes theoretical links between quantization perturbations, Fisher information, and loss-landscape sharpness, then introduces a Fisher-aware mixed-precision scheme and Fisher-trace regularization to mitigate the resulting gaps. Empirical results demonstrate improved critical-category mAP under PTQ and QAT across multiple DETR variants and datasets, with notable gains on challenging categories while keeping overall mAP stable. The findings highlight the importance of geometry-aware quantization for safety-critical deployment of object detectors in real-world applications.

Abstract

The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset.

Figures (8)

• Figure 1: Overview. We investigate a practical setting with task-dependent critical-category objectives in \ref{['ssec:form']}. We empirically observe disparate effects of quantization on the critical-category performance in \ref{['ssec:analy']}, where post-training quantization (PTQ) and quantization-aware training (QAT) lead to performance gaps for critical categories w.r.t. to a floating-point (FP) model. We theoretically analyze such gaps using Fisher information framework and propose a Fisher-aware mixed-precision quantization scheme with regularization in \ref{['sec:method']} to overcome these gaps for DETR models.
• Figure 2: Bit precision vs. layer-wise sensitivity for DETR-R50 on COCO panoptic dataset. There is a clear correlation between the number of bits and our sensitivity metric.
• Figure 3: Bit precision vs. layer-wise sensitivity for DETR-R50, DETR-R101, DAB DETR-R50 and Deformable DETR-R50 on COCO detection dataset, respectively.
• Figure 4: Bit precision vs. layer-wise sensitivity for DETR-R101 on COCO panoptic dataset.
• Figure 5: Comparison of Fisher-critical and Fisher-overall assignments for DETR-R50 on COCO detection dataset when applied to the person category. Our critical objective leads to a significant change in the precision assigned to detector's layers.