Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting

TL;DR

CAT-V tackles the challenge of generating fine-grained, object-centric captions for videos without additional training data. It combines a SAMURAI-based Segmenter, TRACE-Uni temporal analysis, and InternVL-2.5 captioning to produce temporally coherent descriptions guided by user prompts and Chain-of-Thought reasoning. The approach enables precise, attribute-, action-, and context-rich captions for user-selected objects while maintaining temporal continuity and supporting interactive object-based dialogue. This training-free framework reduces reliance on annotated data and enhances controllability for detailed video understanding in practical applications.

Abstract

We present CAT-V (Caption AnyThing in Video), a training-free framework for fine-grained object-centric video captioning that enables detailed descriptions of user-selected objects through time. CAT-V integrates three key components: a Segmenter based on SAMURAI for precise object segmentation across frames, a Temporal Analyzer powered by TRACE-Uni for accurate event boundary detection and temporal analysis, and a Captioner using InternVL-2.5 for generating detailed object-centric descriptions. Through spatiotemporal visual prompts and chain-of-thought reasoning, our framework generates detailed, temporally-aware descriptions of objects' attributes, actions, statuses, interactions, and environmental contexts without requiring additional training data. CAT-V supports flexible user interactions through various visual prompts (points, bounding boxes, and irregular regions) and maintains temporal sensitivity by tracking object states and interactions across different time segments. Our approach addresses limitations of existing video captioning methods, which either produce overly abstract descriptions or lack object-level precision, enabling fine-grained, object-specific descriptions while maintaining temporal coherence and spatial accuracy. The GitHub repository for this project is available at https://github.com/yunlong10/CAT-V

Figures (7)

• Figure 1: Comparison of video captioning approaches: Vanilla (top-left), Dense (top-right), Dense Object (bottom-left), and our CAT-V framework (bottom-right) with integrated modules for user-controlled object-centric captioning via integrated modules (Segmenter, Temporal Analyzer, Captioner with CoT reasoning).
• Figure 2: CAT-V consists of three modules: Segmenter, Temporal Analyzer, and Captioner. The Segmenter precisely segments objects in video frames using user-defined prompts (points, bounding boxes, or regions). The Temporal Analyzer captures video dynamics hierarchically. The Captioner creates object-centric captions using upstream information and CoT reasoning.
• Figure 3: CAT-V can focus on different objects within the same video. The top sequence shows object-centric captioning for a horse, while the bottom sequence demonstrates captioning for the cowboy, each with precise temporal segmentation of their respective actions and states.
• Figure 4: Examples of CAT-V's support for various visual prompting formats. The system effectively handles points, bounding boxes, and irregular regions to identify and track diverse objects including pandas, birds, bottles, and people, demonstrating its flexibility and accuracy in accommodating different user input preferences.
• Figure 5: Comparison of different visual prompt styles (Bounding Box, Blur, Circle, Color Block, Halo, Mask, and Polygon) for highlighting a blue plastic cup, demonstrating their effects on object-centric captioning accuracy.