LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models

TL;DR

This work targets the practical deployment of large multimodal models by reducing both memory and compute via a joint low-rank plus quantization compression in the frequency domain. By transforming weight matrices to the frequency domain, LLaVA-FA exploits de-correlation and conjugate symmetry to achieve equivalent or better reconstruction with fewer parameters, embodied in a frequency-domain low-rank plus quantization scheme. The authors introduce PolarQuant for complex-valued weights and an optional diagonal calibration (ODC) to avoid large calibration data, achieving strong performance across vision-language benchmarks with much lower activation and latency costs. Empirical results demonstrate that LLaVA-FA outperforms existing efficient LMMs on multiple tasks, with substantial reductions in training data, parameters, and inference overhead, enabling more accessible deployment of multimodal AI systems.

Abstract

Large multimodal models (LMMs) have achieved impressive performance on various vision-language tasks, but their substantial computational and memory costs hinder their practical deployment. Existing compression methods often decouple low-rank decomposition and quantization, leading to compounded reconstruction errors, especially in multimodal architectures with cross-modal redundancy. To address this issue, we propose LLaVA-FA, a novel efficient LMM that performs joint low-rank plus quantization approximation in the frequency domain. By leveraging the de-correlation and conjugate symmetry properties of Fourier transform, LLaVA-FA achieves more compact and accurate weight representations. Furthermore, we introduce PolarQuant, a polar-coordinate quantization method tailored for complex matrices, and an optional diagonal calibration (ODC) scheme that eliminates the need for large-scale calibration data. Extensive experimental results demonstrate that our proposed LLaVA-FA outperforms existing efficient multimodal models across multiple benchmarks while maintaining minimal activated parameters and low computational costs, validating its effectiveness as a powerful solution for compressing LMMs.

Figures (8)

• Figure 1: Comparisons of training cost and performance. LLaVA-FA achieves comparable performance with advanced LMMs using lower training costs while outperforming current efficient LMMs by a large margin.
• Figure 2: The architecture of the proposed LLaVA-FA.
• Figure 3: Illustration of Fourier approximation. Left: Singular value spread in the spatial domain vs. frequency domain. Right: LLaVA-FA decomposes a real-valued weight matrix $\widetilde{\mathbf{W}}$ into a frequency-domain low-rank component ($\widetilde{\mathbf{L}}_1$ and $\widetilde{\mathbf{L}}_2$) that keeps the top-r singular values with $b_r$, $b_\theta$ bits, plus a PolarQuant residual $\widetilde{\mathbf{Q}}$ for the remaining spectrum. The de-correlation and conjugate symmetry of Fourier transform lets us store only half of the complex coefficients, yielding the same rank as spatial-domain truncation but with fewer parameters and smaller Frobenius error.
• Figure 4: Latency and KV cache usage of LLaVA-FA.
• Figure 5: Qualitative comparison. LLaVA-FA-2B demonstrates superior precision and efficiency on fine-grained recognition (left) and OCR tasks (right). It correctly identifies specific details (e.g., "petunias") and prices with the lowest latency among all baselines.

Theorems & Definitions (9)

• Theorem B.1
• proof
• Theorem B.2
• proof
• Proposition 1
• proof
• Proposition 2
• proof
• Lemma B.3