Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation

Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang

TL;DR

This work broadens Referring Expression understanding and generation from single-target to generalized tasks that include multi-target and no-target expressions (GRES, GREC, GREG). It introduces gRefCOCO, a large-scale dataset with rich annotations (masks and boxes) and diverse expressions, and proposes ReLA, a region-based baseline that explicitly models region-region and region-language interactions. ReLA achieves state-of-the-art results on GRES and GREC, demonstrates strong performance on classic RES/REC benchmarks when adapted, and extends to RVOS with competitive results. The GREx framework and gRefCOCO pave the way for more flexible, robust, and real-world applicable grounding, comprehension, and generation across images and videos.

Abstract

Referring Expression Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new benchmarks called Generalized Referring Expression Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-language dependencies. The proposed ReLA achieves the state-of-the-art results on the both GRES and GREC tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GREx.

GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation

TL;DR

This work broadens Referring Expression understanding and generation from single-target to generalized tasks that include multi-target and no-target expressions (GRES, GREC, GREG). It introduces gRefCOCO, a large-scale dataset with rich annotations (masks and boxes) and diverse expressions, and proposes ReLA, a region-based baseline that explicitly models region-region and region-language interactions. ReLA achieves state-of-the-art results on GRES and GREC, demonstrates strong performance on classic RES/REC benchmarks when adapted, and extends to RVOS with competitive results. The GREx framework and gRefCOCO pave the way for more flexible, robust, and real-world applicable grounding, comprehension, and generation across images and videos.

Abstract

Referring Expression Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new benchmarks called Generalized Referring Expression Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-language dependencies. The proposed ReLA achieves the state-of-the-art results on the both GRES and GREC tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GREx.
Paper Structure (20 sections, 5 equations, 15 figures, 16 tables)

This paper contains 20 sections, 5 equations, 15 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Task Setting and Dataset
  4. GREx Task Settings
  5. Evaluation Metrics for GRES
  6. Evaluation Metrics for GREC
  7. Evaluation Metrics for GREG
  8. gRefCOCO: A Large-scale GREx Dataset
  9. Dataset Annotation Procedure and Partitioning
  10. The Proposed Baseline Method ReLA
  11. Architecture Overview
  12. ReLAtionship Modeling
  13. Experiments and Discussion
  14. Implementation Details
  15. Ablation Study
  16. ...and 5 more sections

Figures (15)

  • Figure 1: Classic Referring Expression Segmentation (RES), Comprehension (REC), and Generation (REG), collectively denoted as REx, only supports expressions that indicate a single target object, e.g., "The kid in red". Compared with REx, the proposed Generalized Referring Expression tasks (GREx), including Generalized RES (GRES), Generalized REC (GREC), and Generalized REG (GREG), extend expressions to multi-target or no-target. For example, GREx support multi-target expressions that indicate several objects by their commonalities or relationships, e.g., category (2) "All people", attribute (3) "Standing people", counting (4) "Two people on the far left", and compound (5) "Everyone except the kid in white". GRES and GREC further support no-target expressions that do not match any object, e.g., (6) "The kid in blue".
  • Figure 2: More applications of GREx brought by supporting multi-target and no-target expressions.
  • Figure 3: Examples of the proposed gRefCOCO dataset.
  • Figure 4: Word clouds (top 100 words) and normalized frequency histograms (top 25 words) for expressions in gRefCOCO and RefCOCO.
  • Figure 5: The screenshots of the developed annotation system used for building gRefCOCO. (Kindly zoom in).
  • ...and 10 more figures