FlexCap: Describe Anything in Images in Controllable Detail

TL;DR

FlexCap tackles the need for controllable, region-level image descriptions by introducing length-conditioned localized captions that describe arbitrary image regions with variable information density. It trains on a large-scale, automatically generated image-box-caption dataset built from WebLI and YFCC100M, enabling a 590M-parameter architecture to produce precise, region-specific descriptions. By integrating with an LLM, FlexCap-LLM achieves strong zero-shot VQA and visual dialog performance, while also improving dense captioning and open-ended object detection through a localize-then-describe paradigm. The work demonstrates that human-interpretable, region-focused text combined with powerful reasoning in LLMs can achieve competitive results across multiple benchmarks, with practical implications for accessibility and interactive AI, albeit with caveats around data biases and non-end-to-end training.

Abstract

We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input boxes, enabling control over information density, with descriptions ranging from concise object labels to detailed captions. To achieve this, we create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap's effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap's localized descriptions can serve as input to a large language model to create a visual question answering (VQA) system, achieving state-of-the-art zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap's utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex-cap.github.io .

Figures (14)

• Figure 1: FlexCap generates controllably rich localized descriptions for any region in an image as shown on the left. It has the flexibility to produce captions in a controllable manner which allows the full spectrum of valid descriptions to be explored from short object category names to fully-detailed captions. On the right, we demonstrate that rich localized captions generated by FlexCap, when coupled with large language models (LLMs), enable zero-shot visual question answering.
• Figure 2: Architecture and Training Setup. FlexCap outputs a length-controlled caption of the object contained in the bounding box by taking (left) an image, (middle) coordinates of a bounding box and (right) the length prefix and caption, as inputs. The training loss is the standard next-word prediction loss that is used to train image captioning models.
• Figure 3: Dataset Generation. We use OWL-ViT to generate a dataset of triplets of image, bounding box and captions from a web-scale dataset of noisy image-text pairs. Increasing levels of richness in captions is captured through different length descriptions for each box.
• Figure 4: Evaluating open-vocabulary outputs from FlexCap using the CLIP radford2021learning text encoder.
• Figure 5: Examples of length controlled captions generated by FlexCap. Note that attributes ("pink flamingo kite") and context ("in the jungle") are generated as the length increases.