Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuro-Symbolic Spatial Reasoning in Segmentation

Jiayi Lin, Jiabo Huang, Shaogang Gong

TL;DR

This work introduces RelateSeg, a neuro-symbolic approach for open-vocabulary semantic segmentation that explicitly reasons about spatial relationships. By extracting spatial triplets from VLMs, introducing pseudo categories to connect spatial regions, and enforcing these relations through first-order logic with fuzzy relaxation, RelateSeg integrates structured knowledge into a neural segmentation model. The method achieves state-of-the-art average mIoU across four OVSS benchmarks, with notable gains on images containing multiple categories, while requiring only a single auxiliary loss and no extra parameters. The approach demonstrates the value of combining perceptual learning with symbolic spatial reasoning, offering improved scene coherence and a pathway toward scalable, relational OVSS, though it faces limits in highly complex scenes and incurs additional inference time.

Abstract

Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of categories, requiring generalization to unseen and unlabelled objects. Using vision-language models (VLMs) to correlate local image patches with potential unseen object categories suffers from a lack of understanding of spatial relations of objects in a scene. To solve this problem, we introduce neuro-symbolic (NeSy) spatial reasoning in OVSS. In contrast to contemporary VLM correlation-based approaches, we propose Relational Segmentor (RelateSeg) to impose explicit spatial relational constraints by first order logic (FOL) formulated in a neural network architecture. This is the first attempt to explore NeSy spatial reasoning in OVSS. Specifically, RelateSeg automatically extracts spatial relations, e.g., <cat, to-right-of, person>, and encodes them as first-order logic formulas using our proposed pseudo categories. Each pixel learns to predict both a semantic category (e.g., "cat") and a spatial pseudo category (e.g., "right of person") simultaneously, enforcing relational constraints (e.g., a "cat" pixel must lie to the right of a "person"). Finally, these logic constraints are formulated in a deep network architecture by fuzzy logic relaxation, enabling end-to-end learning of spatial-relationally consistent segmentation. RelateSeg achieves state-of-the-art performance in terms of average mIoU across four benchmark datasets and particularly shows clear advantages on images containing multiple categories, with the cost of only introducing a single auxiliary loss function and no additional parameters, validating the effectiveness of NeSy spatial reasoning in OVSS.

Neuro-Symbolic Spatial Reasoning in Segmentation

TL;DR

This work introduces RelateSeg, a neuro-symbolic approach for open-vocabulary semantic segmentation that explicitly reasons about spatial relationships. By extracting spatial triplets from VLMs, introducing pseudo categories to connect spatial regions, and enforcing these relations through first-order logic with fuzzy relaxation, RelateSeg integrates structured knowledge into a neural segmentation model. The method achieves state-of-the-art average mIoU across four OVSS benchmarks, with notable gains on images containing multiple categories, while requiring only a single auxiliary loss and no extra parameters. The approach demonstrates the value of combining perceptual learning with symbolic spatial reasoning, offering improved scene coherence and a pathway toward scalable, relational OVSS, though it faces limits in highly complex scenes and incurs additional inference time.

Abstract

Open-Vocabulary Semantic Segmentation (OVSS) assigns pixel-level labels from an open set of categories, requiring generalization to unseen and unlabelled objects. Using vision-language models (VLMs) to correlate local image patches with potential unseen object categories suffers from a lack of understanding of spatial relations of objects in a scene. To solve this problem, we introduce neuro-symbolic (NeSy) spatial reasoning in OVSS. In contrast to contemporary VLM correlation-based approaches, we propose Relational Segmentor (RelateSeg) to impose explicit spatial relational constraints by first order logic (FOL) formulated in a neural network architecture. This is the first attempt to explore NeSy spatial reasoning in OVSS. Specifically, RelateSeg automatically extracts spatial relations, e.g., <cat, to-right-of, person>, and encodes them as first-order logic formulas using our proposed pseudo categories. Each pixel learns to predict both a semantic category (e.g., "cat") and a spatial pseudo category (e.g., "right of person") simultaneously, enforcing relational constraints (e.g., a "cat" pixel must lie to the right of a "person"). Finally, these logic constraints are formulated in a deep network architecture by fuzzy logic relaxation, enabling end-to-end learning of spatial-relationally consistent segmentation. RelateSeg achieves state-of-the-art performance in terms of average mIoU across four benchmark datasets and particularly shows clear advantages on images containing multiple categories, with the cost of only introducing a single auxiliary loss function and no additional parameters, validating the effectiveness of NeSy spatial reasoning in OVSS.
Paper Structure (23 sections, 13 equations, 7 figures, 6 tables)

This paper contains 23 sections, 13 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Pseudo Category Based Knowledge Representation
  5. FOL Incorporation for Network Optimization
  6. Experiments
  7. Experimental Setup
  8. Comparative Evaluation on OVSS
  9. Ablation Studies
  10. Further Analysis
  11. Conclusion
  12. Limitations
  13. Correctness of extracted categories through VLMs
  14. Simpler Relation Set
  15. Implementation Details of Spatial Knowledge Acquisition
  16. ...and 8 more sections

Figures (7)

  • Figure 1: Motivation illustration of RelateSeg. The baseline model InvSeg lin2025invseg (third column), without incorporating spatial knowledge, can be confused between categories with similar features, such as zebra vs. giraffe and cat vs. bear, resulting in segmenting one object into two classes due to insufficient understanding of relative object positions. In contrast, RelateSeg (fourth column) achieves clear separation of objects in different spatial positions by leveraging spatial knowledge.
  • Figure 2: Overall framework of RelateSeg. Given a baseline segmentation model, e.g., InvSeg lin2025invseg, we apply spatial constraints to the output probability maps $\{M_{c_i}\}$. A VLM is used to extract a spatial triplet set $\mathbf{T}$ from the input image, based on a category set $\mathbf{C}$ and a fundamental relation set $\mathbf{R}$. These triplets are represented in First-Order Logic and incorporated into optimization, enabling a context-aware neuro-symbolic representation for segmentation.
  • Figure 3: RelateSeg VS Baseline by category number per image on VOC12 and COCO.
  • Figure 4: Examples of Segmentation on COCO (top), VOC12 (middle) and Context (bottom). Each row (a group of five images) is organized as follows from left to right: input image, ground truth (GT), RelateSeg, InvSeg (our implementation), and the spatial graph. In the spatial graph, we use the following notations: "A" = above, "B" = below, "R" = to the right of, "L" = to the left of.
  • Figure 5: RelateSeg VS Baseline in mIOU(%) across constraint Quantity (upper section) and constraint-to-category ratio (lower section) per image on VOC12 and COCO.
  • ...and 2 more figures