FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detection

TL;DR

FQ-PETR is proposed, a fully quantized framework for PETRs, featuring three key innovations, including Quantization-Friendly LiDAR-ray Position Embedding, which approximates complex non-linear functions using two cascaded linear LUTs, achieving high fidelity with minimal entries and no specialized hardware.

Abstract

Camera-based multi-view 3D detection is crucial for autonomous driving. PETR and its variants (PETRs) excel in benchmarks but face deployment challenges due to high computational cost and memory footprint. Quantization is an effective technique for compressing deep neural networks by reducing the bit width of weights and activations. However, directly applying existing quantization methods to PETRs leads to severe accuracy degradation. This issue primarily arises from two key challenges: (1) significant magnitude disparity between multi-modal features-specifically, image features and camera-ray positional embeddings (PE), and (2) the inefficiency and approximation error of quantizing non-linear operators, which commonly rely on hardware-unfriendly computations. In this paper, we propose FQ-PETR, a fully quantized framework for PETRs, featuring three key innovations: (1) Quantization-Friendly LiDAR-ray Position Embedding (QFPE): Replacing multi-point sampling with LiDAR-prior-guided single-point sampling and anchor-based embedding eliminates problematic non-linearities (e.g., inverse-sigmoid) and aligns PE scale with image features, preserving accuracy. (2) Dual-Lookup Table (DULUT): This algorithm approximates complex non-linear functions using two cascaded linear LUTs, achieving high fidelity with minimal entries and no specialized hardware. (3) Quantization After Numerical Stabilization (QANS): Performing quantization after softmax numerical stabilization mitigates attention distortion from large inputs. On PETRs (e.g. PETR, StreamPETR, PETRv2, MV2d), FQ-PETR under W8A8 achieves near-floating-point accuracy (1% degradation) while reducing latency by up to 75%, significantly outperforming existing PTQ and QAT baselines.

Figures (9)

• Figure 1: (a) Illustration of PETR pipeline. The camera-ray position embedding and image features are fused via element-wise addition, and then combined with object queries through a Transformer decoder to generate 3D object predictions. (b) The activation values of camera-ray position embedding feature range form -130 to 130, where image feature are primarily centered around 0.
• Figure 2: Softmax input range and quantization effects. (a) Logits in PETR’s cross-attention span an extremely wide interval, forcing a large per-tensor INT8 scale. (b) With a representative scale $s=5$, rounding the inputs to $\exp(\cdot)$ introduces pronounced piece-wise errors $\lvert\exp(x)-\exp(\hat{x})\rvert$. (c) Softmax outputs from quantized inputs (orange) exhibit peak attenuation and positional shift compared with the floating-point baseline (blue), indicating attention distortion.
• Figure 3: The overall architecture comparison of camera-ray PE, lidar-ray PE and our QFPE.
• Figure 4: Distribution before and after inverse-sigmoid.
• Figure 5: The activation values of QFPE feature range form -29.7 to 29.7.

Theorems & Definitions (2)

• Theorem 1: Anchor Embedding Magnitude Bound
• proof