Connecting Vision and Language with Localized Narratives

TL;DR

This paper introduces Localized Narratives, a scalable multimodal annotation framework that grounds every word of an image caption to a specific image region via synchronized spoken narration and mouse traces. By aligning automatic and manual transcriptions, the authors produce timestamped word-grounded narratives across 848k images from COCO, Flickr30k, ADE20K, and Open Images, accompanied by extensive quality analyses. They show the data are rich, diverse, and accurate, enabling dense grounding beyond previous datasets. As a key application, they demonstrate controlled image captioning where the generated caption adheres to the sequence and regions indicated by the mouse trace, highlighting potential for enhanced grounding in generation, retrieval, and assistive technologies.

Abstract

We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.

Figures (12)

• Figure 1: Localized Narrative example: Caption, voice, and mouse trace synchronization represented by a color gradient . The project website locnarr-website contains a visualizer with many live examples.
• Figure 2: Sample annotations from (a) COCO Captions chen15arxiv, (b) Flickr30k Entities plummer17ijcv, (c) Visual Genome krishna17ijcv, and (d) Localized Narratives (Ours). For clarity, (b) shows a subset of region descriptions and (d) shows a shorter-than-average Localized Narrative.
• Figure 3: Localized Narratives annotation: We align the automatic transcription (c) to the manual one (d) to transfer the timestamps from the former to the latter, resulting in a transcription that is both accurate and timestamped (e). To do so, we perform a sequence-to-sequence alignment (gray box) between $a_i$ and $m_j$ (black thick lines). The timestamps of matched words $m_j$ are defined as the segment (green) containing the original timestamps (red) of the matched words $a_i$. Unmatched words $m_j$ get assigned the time segments in between matched neighboring words (blue). These timestamps are transferred to the mouse trace and define the trace segment for each word $m_j$.
• Figure 4: Mouse trace segment locations on COCO with respect to the closest box of the relevant class (\ref{['fig:location_hist:box']}).
• Figure 5: Distribution of number of nouns per caption. As in Table \ref{['tab:richness']}, these counts are per individual caption.