NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation

TL;DR

NICEST reframes scene graph generation as a noisy-label learning problem to address two flawed dataset assumptions: that all positive annotations are equally correct and that all unannotated negatives are background. It introduces NICE to detect and softly correct noisy labels through Neg-NSD, Pos-NSD, and NSC, and NIST, a multi-teacher knowledge distillation framework, to learn unbiased fusion knowledge and mitigate tail-head bias. A new VG-OOD benchmark is proposed to measure out-of-distribution generalization beyond frequency priors. Across VG, VG-OOD, and GQA, NICEST consistently improves mean and tail-predicate performance while maintaining or boosting head predicates, demonstrating strong generalization and robustness to noisy supervision.

Abstract

Nearly all existing scene graph generation (SGG) models have overlooked the ground-truth annotation qualities of mainstream SGG datasets, i.e., they assume: 1) all the manually annotated positive samples are equally correct; 2) all the un-annotated negative samples are absolutely background. In this paper, we argue that neither of the assumptions applies to SGG: there are numerous noisy ground-truth predicate labels that break these two assumptions and harm the training of unbiased SGG models. To this end, we propose a novel NoIsy label CorrEction and Sample Training strategy for SGG: NICEST. Specifically, it consists of two parts: NICE and NIST, which rule out these noisy label issues by generating high-quality samples and the effective training strategy, respectively. NICE first detects noisy samples and then reassigns them more high-quality soft predicate labels. NIST is a multi-teacher knowledge distillation based training strategy, which enables the model to learn unbiased fusion knowledge. And a dynamic trade-off weighting strategy in NIST is designed to penalize the bias of different teachers. Due to the model-agnostic nature of both NICE and NIST, our NICEST can be seamlessly incorporated into any SGG architecture to boost its performance on different predicate categories. In addition, to better evaluate the generalization of SGG models, we further propose a new benchmark VG-OOD, by re-organizing the prevalent VG dataset and deliberately making the predicate distributions of the training and test sets as different as possible for each subject-object category pair. This new benchmark helps disentangle the influence of subject-object category based frequency biases. Extensive ablations and results on different backbones and tasks have attested to the effectiveness and generalization ability of each component of NICEST.

Figures (10)

• Figure 1: An illustration of three types of noisy annotations in SGG datasets, taking VG as an example. (a) Common-prone: For some triplets, the annotators tend to select less informative coarse-grained predicates (brown) instead of the fine-grained ones (green). The subject and object for each triplet are denoted by blue and pink boxes, respectively. (b) Synonym-random: For some triplets, annotators usually randomly choose one predicate from the several synonyms (e.g., has and with are synonyms for $\langle$man/woman-shirt$\rangle$). Original: The $t$-SNE visualization of original triplets $\langle$man-has/with-shirt$\rangle$ features. For brevity, we randomly sample parts of triplets for each type. New: The $t$-SNE visualization of same triplets after NICE. (c) Negative: Some negative triplets may not be background (the green dash arrows).
• Figure 2: The triplets wrongly changed by NICE. The original predicates are in brown and the changed predicates are in red.
• Figure 3: The pipeline of NICE (taking an image from VG as an example). (a) Neg-NSD: Given all negative triplets (blue dash arrows), the OOD detection model detects missing annotated triplets ($\mathcal{T}^-_{\text{noisy}}$) and assigns pseudo labels to them (green predicates). (b) Pos-NSD: Given the newly composed positive triplet set ($\widetilde{\mathcal{T}}^+$), Pos-NSD detects all noisy positive samples ($\widetilde{\mathcal{T}}^+_{\text{noisy}}$). (c) NSC: NSC reassigns more high-quality soft predicate labels to all noisy positive samples, and the black and red predicates are the scores of the original and raw predicate categories in soft labels. Finally, we obtain a new cleaner version of ground-truth annotations.
• Figure 4: Left: Multidimensional scaling visualization of features of randomly sampled triplets with predicate in. Right: Detected clean samples and noisy samples by Pos-NSD.
• Figure 5: Above: The multidimensional scaling visualization of features of randomly sampled triplets with predicate in with cutoff distance ranked at 50% (left) and 1% (right). Below: The triplet categories of the randomly sampled visual relation triplets from the corresponding red circle and green circle.