Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

TL;DR

This work introduces HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen and suggests that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings.

Abstract

Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.

Figures (8)

• Figure 1: Overview of the HDFLIM algorithm where Image Encoder and LLM are kept frozen. HD representations of the $i^{th}$ token are binded with the HD representation of the image, and this acts as an contextual cue for the next token during the inference stage.
• Figure 2: Captioning performance of HDFLIM variants evaluated against the ZeroCap baseline (dotted line). HDFLIM show transferability from Base to Instruct Qwen4B variants, with modest performance degradation, when both variants are compared to ZeroCap.
• Figure 3: Reference free metric evaluation on COCO. Max number of generated tokens for HDFLIM (P) are 41 with no prompt for paraphrasing. For Qwen2VL$_{\text{Base}}$; max generated tokens are 50 with prompt: "Provide a detailed description of this image, including the main subjects, their actions, the setting, and any notable details."
• Figure 4: Inference Speed vs. Caption Length. These experiments were conducted on an NVIDIA A100 (40GB) GPU. ZeroCap and ConZIC were re-implemented for this study using the original hyperparameters reported in their respective publications to ensure fair benchmarking.
• Figure 5: HDFLIM (P) generated Long Caption: "Boy sleeping on a bed in his room, which is empty except for the mosquito net and some furniture. The background has an old wall with peeling paint that looks like a faded blue color."