A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generation

TL;DR

This work tackles spurious correlations in two-stage scene graph generation by reconfiguring the causal structure from $X \rightarrow R \rightarrow Y$ to a reverse form $X \rightarrow R \leftarrow Y$, thereby decoupling input-image signals from final predictions. It introduces Active Reverse Estimation (ARE) to actively manipulate the reverse causal space and Maximum Information Sampling (MIS) to maximize information gain across object pairs, jointly mitigating head-tail and fore-back biases. The authors provide theoretical insights and empirical evidence across VG150, GQA, Open Images V6, and PSG, achieving state-of-the-art mean recall (mR@K) while maintaining competitive recall (R@K) and offering a practical training-time overhead that remains modest. Overall, RcSGG provides a model-agnostic, causality-aware debiasing framework that improves fine-grained relationship prediction and offers a robust alternative to traditional reweighting or reannotation strategies in SGG.

Abstract

Existing two-stage Scene Graph Generation (SGG) frameworks typically incorporate a detector to extract relationship features and a classifier to categorize these relationships; therefore, the training paradigm follows a causal chain structure, where the detector's inputs determine the classifier's inputs, which in turn influence the final predictions. However, such a causal chain structure can yield spurious correlations between the detector's inputs and the final predictions, i.e., the prediction of a certain relationship may be influenced by other relationships. This influence can induce at least two observable biases: tail relationships are predicted as head ones, and foreground relationships are predicted as background ones; notably, the latter bias is seldom discussed in the literature. To address this issue, we propose reconstructing the causal chain structure into a reverse causal structure, wherein the classifier's inputs are treated as the confounder, and both the detector's inputs and the final predictions are viewed as causal variables. Specifically, we term the reconstructed causal paradigm as the Reverse causal Framework for SGG (RcSGG). RcSGG initially employs the proposed Active Reverse Estimation (ARE) to intervene on the confounder to estimate the reverse causality, \ie the causality from final predictions to the classifier's inputs. Then, the Maximum Information Sampling (MIS) is suggested to enhance the reverse causality estimation further by considering the relationship information. Theoretically, RcSGG can mitigate the spurious correlations inherent in the SGG framework, subsequently eliminating the induced biases. Comprehensive experiments on popular benchmarks and diverse SGG frameworks show the state-of-the-art mean recall rate.

Figures (11)

• Figure 1: The motivations of RcSGG. Images and count numbers are sourced from VG150. Test results were obtained from the MotifsNet model under the PredCls mode. (a) Distribution of head and tail relationships. Here, we approximate the top 50% as head relationships (on, has, in, of, wearing), with the remainder classified as tail relationships. (b) Foreground and background relationships in the scene graph generation task and their respective proportions over the entire dataset. (c) The number of right and wrong predictions, where we also showcase the number of instances incorrectly predicted as partial head relationships. (d) The number of right predictions under two scenarios: considering and not considering background relationships.
• Figure 2: The standard pipeline of the two-stage scene graph generation framework, where the input image $X$ is processed by a pre-trained object detector to obtain relationship features $R$, followed by a relationship classifier to produce the final output $Y$.
• Figure 3: (a) Sampling independence of the relationship feature space and the image space. Solid spheres, , , and , denote different relationships. An irregular closed black curve symbolizes an image. (b) Co-occurrences of relationships in the VG150 train set; only a subset is displayed for clarity.
• Figure 4: Structural Causal Model (SCM). (a) is the causal chain structure of the typical two-stage SGG method. (b) is the reverse causal structure we propose.
• Figure 5: (a) Distribution of relationship features across the entire dataset. (b) and (c) depict the distribution of relationship features within a batch; the former represents the original distribution, while the latter illustrates the distribution optimized by our proposed Active Reverse Estimation (ARE).