Beyond Real Weights: Hypercomplex Representations for Stable Quantization

TL;DR

This work addresses the efficiency gap in multimodal large language models by introducing Parameterized Hypercomplex Multiplication (PHM) to replace dense feed-forward layers in a staged, stable manner. A residual transition with an interpolation coefficient, together with reconstruction and knowledge-distillation losses, enables a smooth shift from dense to hypercomplex representations. The approach achieves substantial parameter and FLOPs reductions while preserving multimodal alignment, delivering competitive captioning and QA performance across multiple benchmarks and enabling faster inference. The results suggest a practical, architecture-compatible path toward efficient multimodal reasoning that complements low-bit quantization techniques.

Abstract

Multimodal language models (MLLMs) require large parameter capacity to align high-dimensional visual features with linguistic representations, making them computationally heavy and difficult to deploy efficiently. We introduce a progressive reparameterization strategy that compresses these models by gradually replacing dense feed-forward network blocks with compact Parameterized Hypercomplex Multiplication (PHM) layers. A residual interpolation schedule, together with lightweight reconstruction and knowledge distillation losses, ensures that the PHM modules inherit the functional behavior of their dense counterparts during training. This transition yields substantial parameter and FLOP reductions while preserving strong multimodal alignment, enabling faster inference without degrading output quality. We evaluate the approach on multiple vision-language models (VLMs). Our method maintains performance comparable to the base models while delivering significant reductions in model size and inference latency. Progressive PHM substitution thus offers an architecture-compatible path toward more efficient multimodal reasoning and complements existing low-bit quantization techniques.

Figures (3)

• Figure 1: Comparison of Qwen2.5-VL 7B outputs: (a) original base model vs. (b) base model with our PHM representations. The PHM variant achieves lower FLOPs, faster inference, and reduced parameters while maintaining comparable captioning quality.
• Figure 2: Given an image and text prompt, the image is encoded by a frozen vision encoder and converted into visual tokens, which are combined with text tokens and passed into the transformer. Attention blocks operate normally with light LoRA tuning, while the heavy FFN blocks are progressively replaced by compact PHM layers. Each FFN is reparameterized as a residual mix of its original dense weight and a PHM operator built from a few fixed $2\times2$ bases $H_b$ and small learnable core matrices $A_b$. Early in training, the dense FFN performs most computation, and PHM contributes little, but through a smooth fade-in schedule guided by Knowledge Distillation and a reconstruction loss, the model gradually hands over computation to PHM while keeping internal features and logits close to the original. After this residual adaptation, the dense FFNs are removed, and the PHM-only model is briefly fine-tuned. This reduces parameters and FLOPs while preserving the model's captioning behavior.
• Figure 3: Comprehensive experimental results: (a) Increasing parameters improves CIDEr, from 59.7 at 4.35B to 88.9 at 8.29B. (b) Higher $\alpha$ reduces validation CE, dropping from 15.0 at $\alpha{=}0.0$ to 1.05 at $\alpha{=}1.0$. (c) LoRA placement impacts accuracy: full LoRA (Q/K/V) achieves 85.0%, while no LoRA drops to 66.0%. (d) Raising $K$ (layers with $B_l{=}3$) boosts CIDEr from 35.0 at $K{=}0$ to 85.5 at $K{=}12$, with model size growing moderately from 5.10B to 5.68B. All the experiments are done on Qwen2.5-VL-7B.