Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

With Great Context Comes Great Prediction Power: Classifying Objects via Geo-Semantic Scene Graphs

Ciprian Constantinescu, Marius Leordeanu

TL;DR

This work argues that object recognition benefits from explicit scene context and introduces the Geo-Semantic Contextual Graph (GSCG), a rich, queryable representation built from a single monocular image by fusing metric depth with unified panoptic and material segmentation. A specialized GraphObjectClassifier then reasons over the target object, its immediate neighbors, and global scene context to predict the object’s class. On COCO 2017 train/val splits, the full GSCG model achieves 73.4% accuracy, significantly outperforming context-agnostic CNN baselines (e.g., 53.5%) and a state-of-the-art multimodal LLM (42.3%), demonstrating the power of explicit, structured context for vision. The work also provides ablations and a human-vs-AI riddle study, highlighting the data-structure and reasoning advantages of GSCGs and outlining future directions toward unknown object understanding and unified multimodal annotations.

Abstract

Humans effortlessly identify objects by leveraging a rich understanding of the surrounding scene, including spatial relationships, material properties, and the co-occurrence of other objects. In contrast, most computational object recognition systems operate on isolated image regions, devoid of meaning in isolation, thus ignoring this vital contextual information. This paper argues for the critical role of context and introduces a novel framework for contextual object classification. We first construct a Geo-Semantic Contextual Graph (GSCG) from a single monocular image. This rich, structured representation is built by integrating a metric depth estimator with a unified panoptic and material segmentation model. The GSCG encodes objects as nodes with detailed geometric, chromatic, and material attributes, and their spatial relationships as edges. This explicit graph structure makes the model's reasoning process inherently interpretable. We then propose a specialized graph-based classifier that aggregates features from a target object, its immediate neighbors, and the global scene context to predict its class. Through extensive ablation studies, we demonstrate that our context-aware model achieves a classification accuracy of 73.4%, dramatically outperforming context-agnostic versions (as low as 38.4%). Furthermore, our GSCG-based approach significantly surpasses strong baselines, including fine-tuned ResNet models (max 53.5%) and a state-of-the-art multimodal Large Language Model (LLM), Llama 4 Scout, which, even when given the full image alongside a detailed description of objects, maxes out at 42.3%. These results on COCO 2017 train/val splits highlight the superiority of explicitly structured and interpretable context for object recognition tasks.

With Great Context Comes Great Prediction Power: Classifying Objects via Geo-Semantic Scene Graphs

TL;DR

This work argues that object recognition benefits from explicit scene context and introduces the Geo-Semantic Contextual Graph (GSCG), a rich, queryable representation built from a single monocular image by fusing metric depth with unified panoptic and material segmentation. A specialized GraphObjectClassifier then reasons over the target object, its immediate neighbors, and global scene context to predict the object’s class. On COCO 2017 train/val splits, the full GSCG model achieves 73.4% accuracy, significantly outperforming context-agnostic CNN baselines (e.g., 53.5%) and a state-of-the-art multimodal LLM (42.3%), demonstrating the power of explicit, structured context for vision. The work also provides ablations and a human-vs-AI riddle study, highlighting the data-structure and reasoning advantages of GSCGs and outlining future directions toward unknown object understanding and unified multimodal annotations.

Abstract

Humans effortlessly identify objects by leveraging a rich understanding of the surrounding scene, including spatial relationships, material properties, and the co-occurrence of other objects. In contrast, most computational object recognition systems operate on isolated image regions, devoid of meaning in isolation, thus ignoring this vital contextual information. This paper argues for the critical role of context and introduces a novel framework for contextual object classification. We first construct a Geo-Semantic Contextual Graph (GSCG) from a single monocular image. This rich, structured representation is built by integrating a metric depth estimator with a unified panoptic and material segmentation model. The GSCG encodes objects as nodes with detailed geometric, chromatic, and material attributes, and their spatial relationships as edges. This explicit graph structure makes the model's reasoning process inherently interpretable. We then propose a specialized graph-based classifier that aggregates features from a target object, its immediate neighbors, and the global scene context to predict its class. Through extensive ablation studies, we demonstrate that our context-aware model achieves a classification accuracy of 73.4%, dramatically outperforming context-agnostic versions (as low as 38.4%). Furthermore, our GSCG-based approach significantly surpasses strong baselines, including fine-tuned ResNet models (max 53.5%) and a state-of-the-art multimodal Large Language Model (LLM), Llama 4 Scout, which, even when given the full image alongside a detailed description of objects, maxes out at 42.3%. These results on COCO 2017 train/val splits highlight the superiority of explicitly structured and interpretable context for object recognition tasks.
Paper Structure (34 sections, 1 figure, 2 tables)

This paper contains 34 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Monocular 3D Scene Understanding.
  4. Scene Graphs.
  5. Contextual Object Recognition.
  6. Methodology
  7. Geo-Semantic Contextual Graph (GSCG) Generation
  8. Unified Scene Perception
  9. Procedural Graph Construction
  10. Contextual Object Classification
  11. Experiments
  12. Experimental Setup
  13. Dataset and Task.
  14. Evaluation Metric.
  15. Our Models.
  16. ...and 19 more sections

Figures (1)

  • Figure 1: The Geo-Semantic Contextual Graph (GSCG) Generation Pipeline. From a single RGB image, we (a) estimate metric depth and perform unified segmentation to get panoptic and material masks. (b) These are fused to create per-instance 3D point clouds. (c) Rich attributes (geometry, material, color) are extracted for each instance. (d) This information is encoded into a GSCG, where nodes represent objects and edges represent spatial relationships. This structured representation then feeds our classifier.