Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video

Zhaobo Qi, Yibo Yuan, Xiaowen Ruan, Shuhui Wang, Weigang Zhang, Qingming Huang

TL;DR

Data biases in Temporal Sentence Grounding in Video (TSGV) arise from uneven temporal distributions that enable shortcuts in learning. The authors propose BSSARD, an adversarial debiasing framework that synthesizes bias-conflict samples via a Visual Bias Generator and a Query Bias Generator and trains them against a Bias Discriminator within a span-based grounding model. The approach covers broad spurious correlations between visual/text modalities and temporal targets, and key results on Charades-CD and ActivityNet-CD show improved debiasing and generalization, including out-of-distribution scenarios. Extensive ablations validate the contributions of bias generators, fusion strategies, and training schedules, demonstrating improved robustness of TSGV systems to dataset biases. Overall, BSSARD advances reliable cross-modal grounding by explicitly challenging bias mechanisms during training and enforcing stronger multimodal alignment.

Abstract

Temporal Sentence Grounding in Video (TSGV) is troubled by dataset bias issue, which is caused by the uneven temporal distribution of the target moments for samples with similar semantic components in input videos or query texts. Existing methods resort to utilizing prior knowledge about bias to artificially break this uneven distribution, which only removes a limited amount of significant language biases. In this work, we propose the bias-conflict sample synthesis and adversarial removal debias strategy (BSSARD), which dynamically generates bias-conflict samples by explicitly leveraging potentially spurious correlations between single-modality features and the temporal position of the target moments. Through adversarial training, its bias generators continuously introduce biases and generate bias-conflict samples to deceive its grounding model. Meanwhile, the grounding model continuously eliminates the introduced biases, which requires it to model multi-modality alignment information. BSSARD will cover most kinds of coupling relationships and disrupt language and visual biases simultaneously. Extensive experiments on Charades-CD and ActivityNet-CD demonstrate the promising debiasing capability of BSSARD. Source codes are available at https://github.com/qzhb/BSSARD.

Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video

TL;DR

Data biases in Temporal Sentence Grounding in Video (TSGV) arise from uneven temporal distributions that enable shortcuts in learning. The authors propose BSSARD, an adversarial debiasing framework that synthesizes bias-conflict samples via a Visual Bias Generator and a Query Bias Generator and trains them against a Bias Discriminator within a span-based grounding model. The approach covers broad spurious correlations between visual/text modalities and temporal targets, and key results on Charades-CD and ActivityNet-CD show improved debiasing and generalization, including out-of-distribution scenarios. Extensive ablations validate the contributions of bias generators, fusion strategies, and training schedules, demonstrating improved robustness of TSGV systems to dataset biases. Overall, BSSARD advances reliable cross-modal grounding by explicitly challenging bias mechanisms during training and enforcing stronger multimodal alignment.

Abstract

Temporal Sentence Grounding in Video (TSGV) is troubled by dataset bias issue, which is caused by the uneven temporal distribution of the target moments for samples with similar semantic components in input videos or query texts. Existing methods resort to utilizing prior knowledge about bias to artificially break this uneven distribution, which only removes a limited amount of significant language biases. In this work, we propose the bias-conflict sample synthesis and adversarial removal debias strategy (BSSARD), which dynamically generates bias-conflict samples by explicitly leveraging potentially spurious correlations between single-modality features and the temporal position of the target moments. Through adversarial training, its bias generators continuously introduce biases and generate bias-conflict samples to deceive its grounding model. Meanwhile, the grounding model continuously eliminates the introduced biases, which requires it to model multi-modality alignment information. BSSARD will cover most kinds of coupling relationships and disrupt language and visual biases simultaneously. Extensive experiments on Charades-CD and ActivityNet-CD demonstrate the promising debiasing capability of BSSARD. Source codes are available at https://github.com/qzhb/BSSARD.
Paper Structure (31 sections, 11 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 31 sections, 11 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Sentence Grounding in Video
  4. Model Debiasing
  5. Method
  6. Overview
  7. Bias Generator
  8. Visual Bias Generator
  9. Query Bias Generator
  10. Grounding Model
  11. Training Objectives
  12. Training Process
  13. Bias Generator Training Objectives
  14. Discriminator Training Objectives
  15. Experiment
  16. ...and 16 more sections

Figures (4)

  • Figure 1: The spurious correlation between query words and temporal location of target moments. The horizontal and vertical axes represent the normalized starting time and duration of the target moment, respectively. The color represents the text-video pair density, which is obtained by using kernel density estimation with the Gaussian kernel.
  • Figure 2: An overview of our BSSARD. The orange background is only used during the training period. Best viewed in color.
  • Figure 3: The visualization comparison results between VLSNet* and BSSAR-VLSNet*.
  • Figure 4: The temporal distribution of target moments for video-text samples with certain common verbs. The horizontal and vertical axes denote the normalized starting time and duration of the target moment, respectively. The color represents the sample density.