Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech

Girish A. Koushik, Helen Treharne, Diptesh Kanojia

TL;DR

This work addresses the challenge of detecting hate speech in long-form multimodal content by transforming it into a structured reasoning problem. It introduces TANDEM, a tandem reinforcement learning framework that jointly optimizes vision-language and audio-language models to produce temporally grounded, target-aware predictions with interpretable reasoning. Across HateMM, MultiHateClip, and ImpliHateVid, TANDEM surpasses zero-shot and context-augmented baselines, achieving notable gains in target identification (e.g., $0.73$ F1 on HateMM) and temporal grounding while highlighting the remaining difficulty of distinguishing offensive versus hateful content in multiclass settings. The approach offers a blueprint for transparent and actionable online safety moderation by providing precise timestamps, targets, and rationale, rather than relying on coarse classifications.

Abstract

Social media platforms are increasingly dominated by long-form multimodal content, where harmful narratives are constructed through a complex interplay of audio, visual, and textual cues. While automated systems can flag hate speech with high accuracy, they often function as "black boxes" that fail to provide the granular, interpretable evidence, such as precise timestamps and target identities, required for effective human-in-the-loop moderation. In this work, we introduce TANDEM, a unified framework that transforms audio-visual hate detection from a binary classification task into a structured reasoning problem. Our approach employs a novel tandem reinforcement learning strategy where vision-language and audio-language models optimize each other through self-constrained cross-modal context, stabilizing reasoning over extended temporal sequences without requiring dense frame-level supervision. Experiments across three benchmark datasets demonstrate that TANDEM significantly outperforms zero-shot and context-augmented baselines, achieving 0.73 F1 in target identification on HateMM (a 30% improvement over state-of-the-art) while maintaining precise temporal grounding. We further observe that while binary detection is robust, differentiating between offensive and hateful content remains challenging in multi-class settings due to inherent label ambiguity and dataset imbalance. More broadly, our findings suggest that structured, interpretable alignment is achievable even in complex multimodal settings, offering a blueprint for the next generation of transparent and actionable online safety moderation tools.

TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech

TL;DR

This work addresses the challenge of detecting hate speech in long-form multimodal content by transforming it into a structured reasoning problem. It introduces TANDEM, a tandem reinforcement learning framework that jointly optimizes vision-language and audio-language models to produce temporally grounded, target-aware predictions with interpretable reasoning. Across HateMM, MultiHateClip, and ImpliHateVid, TANDEM surpasses zero-shot and context-augmented baselines, achieving notable gains in target identification (e.g., F1 on HateMM) and temporal grounding while highlighting the remaining difficulty of distinguishing offensive versus hateful content in multiclass settings. The approach offers a blueprint for transparent and actionable online safety moderation by providing precise timestamps, targets, and rationale, rather than relying on coarse classifications.

Abstract

Social media platforms are increasingly dominated by long-form multimodal content, where harmful narratives are constructed through a complex interplay of audio, visual, and textual cues. While automated systems can flag hate speech with high accuracy, they often function as "black boxes" that fail to provide the granular, interpretable evidence, such as precise timestamps and target identities, required for effective human-in-the-loop moderation. In this work, we introduce TANDEM, a unified framework that transforms audio-visual hate detection from a binary classification task into a structured reasoning problem. Our approach employs a novel tandem reinforcement learning strategy where vision-language and audio-language models optimize each other through self-constrained cross-modal context, stabilizing reasoning over extended temporal sequences without requiring dense frame-level supervision. Experiments across three benchmark datasets demonstrate that TANDEM significantly outperforms zero-shot and context-augmented baselines, achieving 0.73 F1 in target identification on HateMM (a 30% improvement over state-of-the-art) while maintaining precise temporal grounding. We further observe that while binary detection is robust, differentiating between offensive and hateful content remains challenging in multi-class settings due to inherent label ambiguity and dataset imbalance. More broadly, our findings suggest that structured, interpretable alignment is achievable even in complex multimodal settings, offering a blueprint for the next generation of transparent and actionable online safety moderation tools.
Paper Structure (22 sections, 3 equations, 2 figures, 5 tables)

This paper contains 22 sections, 3 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio-Visual Hate Speech
  4. Long Video Understanding and Temporal Segmentation
  5. RL for Multimodal Models
  6. Methodology
  7. Long Video Processing
  8. Hate Segment and Target Identification
  9. RL for VL and AL Models
  10. Experimental Setup
  11. Evaluation Metrics
  12. Experimental Results
  13. Baseline Performance
  14. TANDEM
  15. Discussion
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Sample Video Frames from HateMM Dataset das2023hatemm
  • Figure 2: The TANDEM Architecture. The system employs a tandem reinforcement learning strategy where vision and audio models are iteratively updated while conditioning on each other’s context. The Long Video Processor handles segmentation, while the Multi-Task Reward Function computes a composite score based on summary length, output formatting, classification accuracy, segment localization, and target identification to guide the policy gradient update.