PhraseStereo: The First Open-Vocabulary Stereo Image Segmentation Dataset

TL;DR

PhraseStereo addresses the lack of open-vocabulary phrase grounding in stereo imagery by extending the PhraseCut framework to paired left-right views. It constructs a large stereo dataset by generating right-view images with a diffusion-based GenStereo pipeline guided by a monocular depth model, embedding disparity information to improve geometric fidelity. The work analyzes how stereo-based generation quality varies with baseline-like scale factors and demonstrates the dataset's potential to enable models that jointly reason about language, semantics, and geometry, while acknowledging limitations due to hallucinations in synthetic right views. A key future direction is integrating stereo disparity supervision to align generated right views with real stereo geometry, enhancing depth-informed grounding performance.

Abstract

Understanding how natural language phrases correspond to specific regions in images is a key challenge in multimodal semantic segmentation. Recent advances in phrase grounding are largely limited to single-view images, neglecting the rich geometric cues available in stereo vision. For this, we introduce PhraseStereo, the first novel dataset that brings phrase-region segmentation to stereo image pairs. PhraseStereo builds upon the PhraseCut dataset by leveraging GenStereo to generate accurate right-view images from existing single-view data, enabling the extension of phrase grounding into the stereo domain. This new setting introduces unique challenges and opportunities for multimodal learning, particularly in leveraging depth cues for more precise and context-aware grounding. By providing stereo image pairs with aligned segmentation masks and phrase annotations, PhraseStereo lays the foundation for future research at the intersection of language, vision, and 3D perception, encouraging the development of models that can reason jointly over semantics and geometry. The PhraseStereo dataset will be released online upon acceptance of this work.

Figures (3)

• Figure 1: Stereo image pairs with corresponding semantic segmentation masks from PhraseStereo dataset. Each phrase-region pair is color-coded to highlight its linguistic meaning: attributes in blue, categories in red, and relationships in green.
• Figure 2: Overview of the PhraseStereo dataset generation pipeline. Starting from a PhraseCut wu2020phrasecut image $I_L$ (left), we apply a pre-processing step including image resizing. GenStereo qiao2025genstereo, is then used to generate the corresponding right-view image $I_R$ (center). Post-processing includes resizing $I_R$ back to the original resolution of $I_L$, ensuring stereo consistency. The resulting stereo pair ($I_L$, $I_R$), along with the phrase-region annotations pairs, forms the PhraseStereo dataset (right).
• Figure 3: Qualitative analysis of right images in the PhraseStereo dataset across different scale factors. From left to right, each row shows the original left image wu2020phrasecut, the generated right images at $\beta$ increasing. Red rectangles highlight perceptual and geometric inconsistencies. PhraseStereo adopts $\beta = 0.15$ as a balanced configuration.