Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

TL;DR

This work introduces BEST, a framework that bridges vision foundation models and language models via explicit visual clues to produce rich, coherent image-paragraph descriptions without extra cross-modal training. It features a three-stage pipeline for visual clue extraction, candidate synthesis, and candidate selection, plus a novel SPIPE metric that evaluates semantic propositional content through scene-graph comparisons. Empirical results show strong IPC and VQA performance, with domain-adaptive variants and extensive ablations and human evaluation. The approach enables scalable, zero-shot, open-vocabulary image understanding and has broad applications in accessibility, storytelling, and advertisement, while acknowledging limitations in relation reasoning and potential biases.

Abstract

People say, "A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e.g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model to produce a series of comprehensive descriptions for the visual content, which is then verified by the vision model again to select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such a structured semantic representation.

Figures (14)

• Figure 1: Examples of generated image paragraph.
• Figure 2: Framework demonstration. The orange open vocab tagger box corresponds to the image encoder $f_\text{v}(\cdot)$ and the text encoder as $f_\text{t}(\cdot)$. The blue captioner box is the caption model $c(\cdot)$.
• Figure 3: An example of the human-annotated graph and the text extracted graph.
• Figure 4: An example cropped from zeng2022socratic paper, with other outputs for comparison.
• Figure 5: Examples for real-world applications with variants of the proposed framework.