nocaps: novel object captioning at scale

TL;DR

nocaps addresses the limitation of traditional captioning benchmarks by introducing a large-scale, cross-domain evaluation for novel object captioning. It combines COCO caption data with Open Images detections to test models on in-domain, near-domain, and out-of-domain objects, emphasizing the need to separate object recognition from language generation. The authors benchmark two state-of-the-art approaches, CBS and NBT, and show that while leveraging detectors and language models yields substantial gains, human performance remains far superior, especially for out-of-domain objects. The work provides detailed analysis, evaluation protocols, and practical heuristics (e.g., constraint filtering, ELMo integration) to guide future progress toward captioning in the wild.

Abstract

Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these models are to ever function in the wild, a much larger variety of visual concepts must be learned, ideally from less supervision. To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed 'nocaps', for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the OpenImages validation and test sets. The associated training data consists of COCO image-caption pairs, plus OpenImages image-level labels and object bounding boxes. Since OpenImages contains many more classes than COCO, nearly 400 object classes seen in test images have no or very few associated training captions (hence, nocaps). We extend existing novel object captioning models to establish strong baselines for this benchmark and provide analysis to guide future work on this task.

Figures (17)

• Figure 1: The nocaps task setup: Image captioning models must exploit the Open Images object detection dataset (bottom left) to successfully describe novel objects not covered by the COCO Captions dataset (top left). The nocaps benchmark (right) evaluates performance over in-domain, near-domain and out-of-domain subsets of images containing only COCO classes, both COCO and novel classes, and only novel classes, respectively.
• Figure 2: Compared to COCO Captions Chen2015MicrosoftCC, on average nocaps images have more object classes per image (4.0 vs. 2.9), more object instances per image (8.0 vs. 7.4), and longer captions (11 words vs. 10 words). These differences reflect both the increased diversity of the underlying Open Images data openimages, and our image subset selection strategy (refer Section \ref{['sec:data-col']}).
• Figure 3: We conducted pilot studies to evaluate caption collection interfaces. Since Open Images contains rare and fine-grained classes (such as red panda, top right) we found that priming workers with the correct object categories resulted in more accurate and descriptive captions.
• Figure 4: T-SNE maaten2008tsne plots comparing visual (left) and linguistic (right) similarity in COCO, in-domain and out-of-domain classes. We observe that: (a) in-domain shows high visual similarity to COCO (e.g. cat, book (left)). (b) Many out-of-domain classes are visually and linguistically different from in-domain classes (e.g. jellyfish, beetle, cello). (c) out-of-domain also covers many visually and linguistically similar concepts to COCO, which are not well-covered in COCO (e.g. tiger, lemon).
• Figure 5: Examples of near-domain and out-of-domain images from the nocaps validation set. The image on the left belongs to the near-domain subset (COCO and Open Images categories), while the image on the right belongs to out-of-domain subset (only Open Images categories).