SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Data

TL;DR

This work identifies a gap in Referring Expression Segmentation (RES) evaluation, where existing benchmarks fail to test complex, real-world reasoning. It introduces WildRES, a challenging RES benchmark with long, attribute-rich queries and non-distinctive multi-target expressions across diverse domains, and proposes SynRES, a three-step pipeline that automatically generates densely paired image-caption-expression data, refines pseudo-masks via image-text grouping, and applies domain-aware augmentations to bridge synthetic and real data. Empirical results show SynRES consistently improves state-of-the-art RES models (e.g., LISA, GSVA) on WildRES, including strong cross-domain gains on domain-shift subsets, while also delivering competitive improvements on classic RES benchmarks. The approach reduces annotation costs and enhances generalization to real-world scenes, with code and data publicly available for research and downstream applications.

Abstract

Despite the advances in Referring Expression Segmentation (RES) benchmarks, their evaluation protocols remain constrained, primarily focusing on either single targets with short queries (containing minimal attributes) or multiple targets from distinctly different queries on a single domain. This limitation significantly hinders the assessment of more complex reasoning capabilities in RES models. We introduce WildRES, a novel benchmark that incorporates long queries with diverse attributes and non-distinctive queries for multiple targets. This benchmark spans diverse application domains, including autonomous driving environments and robotic manipulation scenarios, thus enabling more rigorous evaluation of complex reasoning capabilities in real-world settings. Our analysis reveals that current RES models demonstrate substantial performance deterioration when evaluated on WildRES. To address this challenge, we introduce SynRES, an automated pipeline generating densely paired compositional synthetic training data through three innovations: (1) a dense caption-driven synthesis for attribute-rich image-mask-expression triplets, (2) reliable semantic alignment mechanisms rectifying caption-pseudo mask inconsistencies via Image-Text Aligned Grouping, and (3) domain-aware augmentations incorporating mosaic composition and superclass replacement to emphasize generalization ability and distinguishing attributes over object categories. Experimental results demonstrate that models trained with SynRES achieve state-of-the-art performance, improving gIoU by 2.0% on WildRES-ID and 3.8% on WildRES-DS. Code and datasets are available at https://github.com/UTLLab/SynRES.

Figures (13)

• Figure 1: (a) Comparison of existing referring expression segmentation (RES) benchmarks with our proposed benchmark, WildRES, which demands advanced complex reasoning and scene understanding. Previous RES datasets yu2016refcocorefcoco+liu2023gres primarily emphasize single-target focus with short queries (1) or distinctive multi-target queries (5) within a similar domain (e.g., images from COCO caesar2018coco). These constraints limit the evaluation of complex queries and generalization ability of RES models. We propose a new benchmark WildRES, which improves single-target expressions with diverse attributes (2)-(4) and refines multi-target ones with shared traits (non-distinctive) and concise phrasing (6)-(8). In addition, WildRES includes in-distribution (col. 2) and domain-shifted subsets (cols. 3, 4) to evaluate generalizability. (b) The state-of-the-art RES method (LISA lai2024lisa) experiences notable performance degradation on our benchmark (highlighted in yellow), which requires advanced reasoning and generalization capability. Our novel synthetic data generation (SynRES) enhances the model's compositional reasoning ability.
• Figure 2: Number of attributions in classic RES datasets vs. WildRES. Using GPT-4o-mini achiam2023gpt4o, we counted the number of attributes in Table \ref{['tab:refer_attr']}. Single-target expressions in WildRES often exceed 11 attributes to specify a target, while multi-target expressions have fewer attributes, similar to classic RES datasets.
• Figure 3: Overview of the step 1 and 2 in SynRES. The process begins by creating distinctive $n$ synthetic expressions for target objects from real images and masks. These expressions are then concatenated and input into a pre-trained text-to-image generative model to produce $m$ synthetic images. Finally, a grouping step is performed to generate reliable synthetic masks by associating appropriate $l$ segmentation masks with their corresponding expressions, yielding densely paired image-mask-expression triplets for objects with diverse attributes, thereby facilitating the learning of distinctive attribute combinations across objects.
• Figure 4: Image-text aligned grouping for reliable pseudo mask generation. Synthetic images and textual expressions are aligned through pseudo mask generation, binary conversion, and pairwise IoU-based clustering to form consensus groups. Final refined segmentation masks are computed via per-group averaging and thresholding, ensuring high-quality mask alignment.
• Figure 5: Augmented examples in the step 3. Mosaic augmentation is applied using synthetic images and masks containing one original real image (blue border) and masks. Specific words (e.g., woman) are replaced with their superclass (e.g., person), a broader concept, with a probability of $p$, which could effectively mitigate model bias toward specific terminology and facilitate the learning of broader vocabulary associations. This substitution process may potentially produce false negative masks (magenta arrows) when other objects belonging to the same superclass exist within the image. Although this frequently exists in human expressions (e.g., RefCOCO+), we manually mitigate this challenge by generating isolated single objects within our synthetic data.