Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Describe Anything: Detailed Localized Image and Video Captioning

Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, Yin Cui

TL;DR

The paper tackles the challenge of generating rich, region-focused captions for images and videos. It introduces Describe Anything Model (DAM) with a focal prompt and a localized vision backbone to preserve fine regional details while maintaining global context, and a semi-supervised DLC-SDP data pipeline to scale high-quality localized captions. To evaluate DLC without relying on reference captions, it proposes DLC-Bench, an attribute-based benchmark with LLM-driven judgments. Empirically, DAM achieves state-of-the-art performance across seven open- and zero-shot benchmarks spanning keyword, phrase, and multi-sentence detailed localization for both images and videos, demonstrating strong generalization and data efficiency. The work advances fine-grained grounded captioning and provides scalable data curation and robust evaluation frameworks for detailed localized vision-language tasks.

Abstract

Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

Describe Anything: Detailed Localized Image and Video Captioning

TL;DR

The paper tackles the challenge of generating rich, region-focused captions for images and videos. It introduces Describe Anything Model (DAM) with a focal prompt and a localized vision backbone to preserve fine regional details while maintaining global context, and a semi-supervised DLC-SDP data pipeline to scale high-quality localized captions. To evaluate DLC without relying on reference captions, it proposes DLC-Bench, an attribute-based benchmark with LLM-driven judgments. Empirically, DAM achieves state-of-the-art performance across seven open- and zero-shot benchmarks spanning keyword, phrase, and multi-sentence detailed localization for both images and videos, demonstrating strong generalization and data efficiency. The work advances fine-grained grounded captioning and provides scalable data curation and robust evaluation frameworks for detailed localized vision-language tasks.

Abstract

Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

Paper Structure

This paper contains 43 sections, 8 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DAM: Describe Anything Model
  4. Task Formulation
  5. Model Architecture
  6. Focal Prompt
  7. Localized Vision Backbone
  8. Extension to Videos
  9. DLC-SDP: SSL-based Data Pipeline
  10. Stage 1: Leveraging Existing Annotations
  11. Stage 2: SSL with Unlabeled Data
  12. DLC-Bench: Benchmark for DLC
  13. Results
  14. Quantitative Results
  15. Qualitative Results
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Describe Anything Model (DAM) generates detailed localized captions for user-specified regions within images (top) and videos (bottom). DAM accepts various region specifications, including clicks, scribbles, boxes, and masks. For videos, specifying the region in any frame suffices.
  • Figure 2: Top: Prior regional captioners derive regional features from global image representations, leading to vague descriptions. Bottom: Zooming in (cropping the image region) enhances detail but loses contextual cues, degrading recognition. This underscores the need for a design that encodes detail-rich regional features while preserving context for improved DLC performance.