Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation

Runfeng Qu, Ole Hall, Pia K Bideau, Julie Ouerfelli-Ethier, Martin Rolfs, Klaus Obermayer, Olaf Hellwich

TL;DR

The paper addresses bias in scene graph generation caused by long-tailed predicate distributions and proposes Salience-SGG, which introduces an Iterative Salience Decoder (ISD) guided by semantic-agnostic bottom-up salience labels. These labels enforce learning of spatially salient triplets, enabling ISD to emphasize spatial structure while remaining robust to debiasing strategies. Across Visual Genome, Open Images V6, and GQA-200, Salience-SGG achieves state-of-the-art results and improves spatial understanding, as evidenced by the new salience-focused evaluation. This work advances unbiased SGG by aligning predictions with salient spatial configurations, and demonstrates strong compatibility with existing debiasing methods and datasets.

Abstract

Scene Graph Generation (SGG) suffers from a long-tailed distribution, where a few predicate classes dominate while many others are underrepresented, leading to biased models that underperform on rare relations. Unbiased-SGG methods address this issue by implementing debiasing strategies, but often at the cost of spatial understanding, resulting in an over-reliance on semantic priors. We introduce Salience-SGG, a novel framework featuring an Iterative Salience Decoder (ISD) that emphasizes triplets with salient spatial structures. To support this, we propose semantic-agnostic salience labels guiding ISD. Evaluations on Visual Genome, Open Images V6, and GQA-200 show that Salience-SGG achieves state-of-the-art performance and improves existing Unbiased-SGG methods in their spatial understanding as demonstrated by the Pairwise Localization Average Precision

Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation

TL;DR

The paper addresses bias in scene graph generation caused by long-tailed predicate distributions and proposes Salience-SGG, which introduces an Iterative Salience Decoder (ISD) guided by semantic-agnostic bottom-up salience labels. These labels enforce learning of spatially salient triplets, enabling ISD to emphasize spatial structure while remaining robust to debiasing strategies. Across Visual Genome, Open Images V6, and GQA-200, Salience-SGG achieves state-of-the-art results and improves spatial understanding, as evidenced by the new salience-focused evaluation. This work advances unbiased SGG by aligning predictions with salient spatial configurations, and demonstrates strong compatibility with existing debiasing methods and datasets.

Abstract

Scene Graph Generation (SGG) suffers from a long-tailed distribution, where a few predicate classes dominate while many others are underrepresented, leading to biased models that underperform on rare relations. Unbiased-SGG methods address this issue by implementing debiasing strategies, but often at the cost of spatial understanding, resulting in an over-reliance on semantic priors. We introduce Salience-SGG, a novel framework featuring an Iterative Salience Decoder (ISD) that emphasizes triplets with salient spatial structures. To support this, we propose semantic-agnostic salience labels guiding ISD. Evaluations on Visual Genome, Open Images V6, and GQA-200 show that Salience-SGG achieves state-of-the-art performance and improves existing Unbiased-SGG methods in their spatial understanding as demonstrated by the Pairwise Localization Average Precision
Paper Structure (9 sections, 3 figures, 9 tables)

This paper contains 9 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiments
  4. Experimental Setup
  5. Comparison to SOTA Methods
  6. Complementary Comparisons
  7. Salience Sensitivity Analysis
  8. Ablation Studies
  9. Conclusion

Figures (3)

  • Figure 1: Salience-SGG. (a): IETrans zhang2022fine with standard debiasing, showing over-reliance on the semantic information, i.e.coat and hanging from (dashed lines). (b): Our salience-enhanced model favors spatially coherent triplets (bold).
  • Figure 2: Comparison of the top-5 detected triplets. For each image, the colors indicate triplet identifications; identical colors before and after re-ranking refer to the same triplets.
  • Figure 3: Pairwise localization avg. precision (pl-AP) versus SGG performance (F@100). Color indicates the model family. The green text marks our contributions