Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension

TL;DR

This paper proposes a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities, and eliminates manual data collection and annotation, enabling scalability and facilitating arbitrary complexity.

Abstract

Referring Expression Comprehension (REC) aims to identify a particular object in a scene by a natural language expression, and is an important topic in visual language understanding. State-of-the-art methods for this task are based on deep learning, which generally requires expensive and manually labeled annotations. Some works tackle the problem with limited-supervision learning or relying on Large Vision and Language Models. However, the development of techniques to synthesize labeled data is overlooked. In this paper, we propose a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities. At first, our pipeline processes existing data to create variations in the annotations. Then, it generates an image using altered annotations as guidance. The result of this pipeline is a new dataset, called Harlequin, made by more than 1M queries. This approach eliminates manual data collection and annotation, enabling scalability and facilitating arbitrary complexity. We pre-train three REC models on Harlequin, then fine-tuned and evaluated on human-annotated datasets. Our experiments show that the pre-training on artificial data is beneficial for performance.

Figures (4)

• Figure 1: Annotations required by the Referring Expression Comprehension task. In this example, the image has one caption with three referring expressions. Each referring expression is accompanied by the location of the referred object (bounding box).
• Figure 2: Our pipeline. It processes existing samples from Flickr30k Entities data. We select the ones characterized by at least one color attribute in their referring expressions. The Annotation Generation Engine processes the sample's caption, referring expressions and locations where the color attribute is replaced with a randomly chosen color. The caption is updated accordingly. Then, the Image Generation Engine creates the new image using new annotations provided by the Annotation Generation Engine as guidance for the generation.
• Figure 3: Examples produced by our pipeline. On the left, we show reference images along with their annotations from Flickr30k Entities. On the right, we report some generated variations. Colors are altered and guide, along with objects' locations, the image synthesis.
• Figure 4: Dataset statistics. We report the number of images and referring expressions per dataset on the left and right, respectively. Harlequin is highlighted in orange.