Unifying Vision-Language Latents for Zero-label Image Caption Enhancement

TL;DR

ViZer addresses the reliance on labeled image-caption data by introducing a zero-label enhancement framework that actively aligns vision and language latent spaces. It inserts a lightweight ViZer mapper between a frozen vision encoder and a language model, training with unlabeled images using a contrastive latent objective and optionally applying LoRA to preserve base capabilities. The approach yields qualitative grounding improvements and consistent gains on SmolVLM-Base and Qwen2-VL, demonstrating a scalable path for label-free multimodal adaptation. This work suggests a practical route to extend zero-label learning beyond captioning to broader vision-language tasks while limiting computational overhead and preserving existing model performance.

Abstract

Vision-language models (VLMs) achieve remarkable performance through large-scale image-text pretraining. However, their reliance on labeled image datasets limits scalability and leaves vast amounts of unlabeled image data underutilized. To address this, we propose Unified Vision-Language Alignment for Zero-Label Enhancement (ViZer), an enhancement training framework that enables zero-label learning in image captioning, providing a practical starting point for broader zero-label adaptation in vision-language tasks. Unlike prior approaches that rely on human or synthetically annotated datasets, ViZer actively aligns vision and language representation features during training, enabling existing VLMs to generate improved captions without requiring text labels or full retraining. We demonstrate ViZer's advantage in qualitative evaluation, as automated caption metrics such as CIDEr and BERTScore often penalize details that are absent in reference captions. Applying ViZer on SmolVLM-Base and Qwen2-VL, we observe consistent qualitative improvements, producing captions that are more grounded and descriptive than their baseline.

Figures (7)

• Figure 1: Qualitative captioning comparison for sample images from OpenImage dataset. We compare ground-truth, baseline, reinforcement-learning, ViZer$_\text{GT}$, and ViZer$_\text{G}$ generated captions. Information that include too much assumption or provide opinionated descriptions are colored in red. Added descriptions by incorporating ViZer or RL are colored in blue.
• Figure 2: Visualization of ViZer mapper width to mapping data size comparison for SmolVLM-Base marafioti2025smolvlm. (right) ViZer$_\text{GT}$ and (left) ViZer$_\text{G}$ both display best results with a width of 256, with a preference for smaller dataset sizes.
• Figure 3: Visual Feature Comparisons on post-training generated image captions for SmolVLM-Base marafioti2025smolvlm (top) and Qwen2-VL Qwen2VL (bottom) with ViZer$_\text{GT}$ (left) and ViZer$_\text{G}$ (right). 2-dimensional PCA is applied to text and vision features for ease of comparison. Points closer to stars represent similar latents.
• Figure 4: GT: Base: RL: ViZer$_{GT}$: ViZer$_{G}$:
• Figure 8: GT: Base: RL: ViZer$_{GT}$: ViZer$_{G}$: