SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers

TL;DR

SemanticDialect is proposed, which advances recent block-wise mixed-format quantization-selecting a per-block optimal format from multiple candidates (a formatbook) by scaling the formatbook with lookup tables for quantization error and quantized values, enabling efficient per-block format selection and quantization at low online cost.

Abstract

Diffusion Transformers (DiT) achieve strong video generation quality, but their memory and compute costs hinder edge deployment. Quantization can reduce these costs, yet existing methods often degrade video quality under high activation variation and the need to preserve semantic/temporal coherence. We propose SemanticDialect, which advances recent block-wise mixed-format quantization-selecting a per-block optimal format (a dialect) from multiple candidates (a formatbook)-by scaling the formatbook with lookup tables for quantization error and quantized values, enabling efficient per-block format selection and quantization at low online cost. We also introduce activation decomposition that reduces quantization error by re-quantizing and adding back residual errors, with attention-guided salient token selection. We further propose semantic-aware dialect assignment (SeDA) to improve quantized value consistency by sharing a sub-formatbook among semantically correlated tokens. Experiments on video DiT (VDiT) models show that SemanticDialect outperforms prior VDiT quantization methods and fine-grained block-wise format baselines, while approaching FP16 quality on Open-Sora 2.0.

Figures (11)

• Figure 1: SemanticDialect is a post-training quantization method for video diffusion transformers. It quantizes Open-Sora 2.0 to 4-bit weights and activations while maintaining near-FP16 visual quality.
• Figure 2: Overview of SemanticDialect. (a) LUT-based efficient online format selection and quantization. (b) Activation decomposition to compensate for quantization-sensitive layers. (c) Semantic-aware dialect assignment (SeDA) to improve quantized-value consistency across blocks.
• Figure 3: Proposed formatbook and lookup table. (a) Four formatbook construction rules. (b) Block-wise normalized value distributions. (c) LUT details and data-to-index conversion.
• Figure 4: Activation decomposition for vector and matrix activations.
• Figure 5: Details of Semantic-Aware Dialect Assignment (SeDA). (a) Anchor/correlated token selection for different attention architectures. (b) Sub-formatbook construction via bin-count profiling. (c) Average anchor-token movement across consecutive denoising timesteps.