One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework

TL;DR

Patch-ioner introduces a patch-centric zero-shot captioning framework that treats image patches as atomic captioning units and aggregates them to describe arbitrary regions, including noncontiguous areas and entire images, without region-level supervision. The approach decouples region specification from image encoding and uses a parameter-free patch aggregation coupled with a text decoder that mitigates modality gaps through memory-based projection or noise training, with DINO-based backbones delivering the strongest local semantics. It unifies multiple captioning tasks—trace, dense, region-set, and image captioning—into a single framework that requires only a single backbone forward pass to caption many regions, enabling scalable, interactive region descriptions. While competitive with state-of-the-art image-captioning models, Patch-ioner excels in local, fine-grained tasks and highlights limitations such as fixed patch context and potential hallucinations from the modality gap, suggesting future work in weak supervision and improved patch-to-text mappings.

Abstract

Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present Patch-ioner, a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense, region-set, and a newly introduced trace captioning task, highlighting the effectiveness of patch-wise semantic representations for scalable caption generation. Project page at https://paciosoft.com/Patch-ioner/ .

Figures (7)

• Figure 1: Patch-ioner: a patch-centric framework for unified zero-shot captioning. A. Overview of our framework. First, we extract language-aligned dense patch embeddings from the image using a VLM. Given a region, we select the underlying patches and aggregate their features to obtain a region representation. Finally, we obtain the region caption by applying a zero-shot text decoder, that is a) conditioned on the latent region representation, b) trained on text-only data, and c) equipped with a mechanism to handle the modality gap present in vision-language common spaces. This enables regional captioning without requiring region-level supervision. B. By aggregating patch-level features from arbitrary image regions, we can flexibly handle multiple captioning tasks across spatial granularities in a unique model.
• Figure 2: Qualitative results from finer (left) to coarser (right) tasks. Note the discrepancy of predicted and ground-truth captions when an image-level (DeCap, DeCap (Crop)) or a CLIP-based regional (CLIP + Mem.) captioner is applied, with respect to our Talk2DINO-based model.
• Figure 3: Memory-based Patch-level captioning. Given an input image, we first extract dense patch-level representations using a vision transformer backbone. For a selected patch, we apply the projection-based mechanism introduced by li2023decap to mitigate the modality gap and align its representation with the text embedding space. Finally, the transformed embedding is fed into a text decoder trained on a text-only corpus, generating a zero-shot caption for the patch.
• Figure 4: LLM Prompt for rephrasing trace captions.
• Figure 5: Narrative vs. Trace Samples. The first column displays sample images from the Localized Narrative dataset pont2020connectingLocalizedNarratives. The remaining three columns show the corresponding mouse traces, along with the captions generated by the LLM. Captions marked with <INVALID> are removed from the dataset.