DenseCap: Fully Convolutional Localization Networks for Dense Captioning

TL;DR

This paper introduces dense captioning, a task that requires localizing multiple regions within an image and describing each region with natural language. It proposes the Fully Convolutional Localization Network (FCLN), an end-to-end architecture that replaces external proposals with a differentiable localization layer based on anchors and bilinear sampling, followed by a region-based recognition network and an LSTM language model. Evaluated on Visual Genome, the approach improves both captioning quality and localization speed over strong baselines, and extends to image retrieval and open-world text grounding. The work enables efficient, region-grounded image understanding with end-to-end trainable components and provides public code/data to accelerate future progress.

Abstract

We introduce the dense captioning task, which requires a computer vision system to both localize and describe salient regions in images in natural language. The dense captioning task generalizes object detection when the descriptions consist of a single word, and Image Captioning when one predicted region covers the full image. To address the localization and description task jointly we propose a Fully Convolutional Localization Network (FCLN) architecture that processes an image with a single, efficient forward pass, requires no external regions proposals, and can be trained end-to-end with a single round of optimization. The architecture is composed of a Convolutional Network, a novel dense localization layer, and Recurrent Neural Network language model that generates the label sequences. We evaluate our network on the Visual Genome dataset, which comprises 94,000 images and 4,100,000 region-grounded captions. We observe both speed and accuracy improvements over baselines based on current state of the art approaches in both generation and retrieval settings.

Figures (5)

• Figure 1: We address the Dense Captioning task (bottom right) by generating dense, rich annotations with a single forward pass.
• Figure 2: Model overview. An input image is first processed a CNN. The Localization Layer proposes regions and smoothly extracts a batch of corresponding activations using bilinear interpolation. These regions are processed with a fully-connected recognition network and described with an RNN language model. The model is trained end-to-end with gradient descent.
• Figure 3: Example captions generated and localized by our model on test images. We render the top few most confident predictions. On the bottom row we additionally contrast the amount of information our model generates compared to the Full image RNN.
• Figure 4: Example image retrieval results using our dense captioning model. From left to right, each row shows a grund-truth test image, ground-truth region captions describing the image, and the top images retrieved by our model using the text of the captions as a query. Our model is able to correctly retrieve and localize people, animals, and parts of both natural and man-made objects.
• Figure 5: Example results for open world detection. We use our dense captioning model to localize arbitrary pieces of text in images, and display the top detections on the test set for several queries.