Enhanced Multimodal Hate Video Detection via Channel-wise and Modality-wise Fusion

TL;DR

This work tackles hate video detection by integrating text, audio, and video through a novel CMFusion architecture that combines temporal cross-attention between video and audio with channel-wise and modality-wise fusion. By extracting modality-specific features (ViT for video, MFCC for audio, BERT for text) and processing temporal dynamics via LSTM and cross-attention, CMFusion produces a unified representation that outperforms five baselines on the HateMM dataset, achieving higher F1 and recall. Ablation studies and case analyses demonstrate the benefits of three-modality integration and advanced fusion, with analysis of feature-space separability supporting improved discriminative power. The approach offers a practical path to more reliable hate content moderation, and the authors provide public code to facilitate adoption and further research.

Abstract

The rapid rise of video content on platforms such as TikTok and YouTube has transformed information dissemination, but it has also facilitated the spread of harmful content, particularly hate videos. Despite significant efforts to combat hate speech, detecting these videos remains challenging due to their often implicit nature. Current detection methods primarily rely on unimodal approaches, which inadequately capture the complementary features across different modalities. While multimodal techniques offer a broader perspective, many fail to effectively integrate temporal dynamics and modality-wise interactions essential for identifying nuanced hate content. In this paper, we present CMFusion, an enhanced multimodal hate video detection model utilizing a novel Channel-wise and Modality-wise Fusion Mechanism. CMFusion first extracts features from text, audio, and video modalities using pre-trained models and then incorporates a temporal cross-attention mechanism to capture dependencies between video and audio streams. The learned features are then processed by channel-wise and modality-wise fusion modules to obtain informative representations of videos. Our extensive experiments on a real-world dataset demonstrate that CMFusion significantly outperforms five widely used baselines in terms of accuracy, precision, recall, and F1 score. Comprehensive ablation studies and parameter analyses further validate our design choices, highlighting the model's effectiveness in detecting hate videos. The source codes will be made publicly available at https://github.com/EvelynZ10/cmfusion.

Figures (3)

• Figure 1: Comparison of feature visualisation with different fusion approaches. Subfigure (a) illustrates the feature visualisation of video samples in the HateMM dataset das2023hatemm, where the features from video, text, and audio are fused using concatenation. In contrast, Subfigure (b) presents the feature visualisation of the same samples where the features from video, text, and audio are integrated through the CMFusion.
• Figure 2: The overview of the CMFusion Model along with the detailed structures of Channel-wise Fusion and Modality-wise Fusion. $F_v$, $F_a$, and $F_t$ represent the input vectors to each module, while $F'_v$, $F'_a$, and $F'_t$ represent the output vectors of the module.
• Figure 3: UMAP visualizations of the feature representations from audio, video, and text modalities, respectively.