Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comment-aided Video-Language Alignment via Contrastive Pre-training for Short-form Video Humor Detection

Yang Liu, Tongfei Shen, Dong Zhang, Qingying Sun, Shoushan Li, Guodong Zhou

TL;DR

This work tackles short-form video humor detection (SVHD) by introducing CVLA, a two-branch hierarchical model that aligns video (vision+audio) and language (title+comments) into a unified semantic space. A data-augmented contrastive pre-training strategy leverages large-scale unlabeled short-form videos to jointly learn video-language alignment and a robust multi-modal fusion representation, then fine-tunes on labeled data for humor detection. The approach yields state-of-the-art performance on DY11k and UR-FUNNY, demonstrating the value of incorporating comments as social-contextual signals and of contrastive pre-training to bridge modality gaps. The expanded DY11k dataset and the CVLA framework offer a generalizable paradigm for multi-modal understanding in short-form media, with potential extension to long-form video tasks.

Abstract

The growing importance of multi-modal humor detection within affective computing correlates with the expanding influence of short-form video sharing on social media platforms. In this paper, we propose a novel two-branch hierarchical model for short-form video humor detection (SVHD), named Comment-aided Video-Language Alignment (CVLA) via data-augmented multi-modal contrastive pre-training. Notably, our CVLA not only operates on raw signals across various modal channels but also yields an appropriate multi-modal representation by aligning the video and language components within a consistent semantic space. The experimental results on two humor detection datasets, including DY11k and UR-FUNNY, demonstrate that CVLA dramatically outperforms state-of-the-art and several competitive baseline approaches. Our dataset, code and model release at https://github.com/yliu-cs/CVLA.

Comment-aided Video-Language Alignment via Contrastive Pre-training for Short-form Video Humor Detection

TL;DR

This work tackles short-form video humor detection (SVHD) by introducing CVLA, a two-branch hierarchical model that aligns video (vision+audio) and language (title+comments) into a unified semantic space. A data-augmented contrastive pre-training strategy leverages large-scale unlabeled short-form videos to jointly learn video-language alignment and a robust multi-modal fusion representation, then fine-tunes on labeled data for humor detection. The approach yields state-of-the-art performance on DY11k and UR-FUNNY, demonstrating the value of incorporating comments as social-contextual signals and of contrastive pre-training to bridge modality gaps. The expanded DY11k dataset and the CVLA framework offer a generalizable paradigm for multi-modal understanding in short-form media, with potential extension to long-form video tasks.

Abstract

The growing importance of multi-modal humor detection within affective computing correlates with the expanding influence of short-form video sharing on social media platforms. In this paper, we propose a novel two-branch hierarchical model for short-form video humor detection (SVHD), named Comment-aided Video-Language Alignment (CVLA) via data-augmented multi-modal contrastive pre-training. Notably, our CVLA not only operates on raw signals across various modal channels but also yields an appropriate multi-modal representation by aligning the video and language components within a consistent semantic space. The experimental results on two humor detection datasets, including DY11k and UR-FUNNY, demonstrate that CVLA dramatically outperforms state-of-the-art and several competitive baseline approaches. Our dataset, code and model release at https://github.com/yliu-cs/CVLA.
Paper Structure (30 sections, 7 equations, 6 figures, 6 tables)

This paper contains 30 sections, 7 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Humor Detection
  4. Video Understanding
  5. Video-Language Pre-training
  6. Methodology
  7. Task Formulation
  8. Architecture
  9. Video Encoding
  10. Language Encoding
  11. Multi-modal Fusion
  12. Data-augmented Contrastive Pre-training
  13. Data Augmentation
  14. Contrastive Tuples Construction
  15. Contrastive Pre-Training
  16. ...and 15 more sections

Figures (6)

  • Figure 1: An example for SVHD.
  • Figure 2: The overall framework of our proposed CVLA for short-form video humor detection.
  • Figure 3: The statistics of our expanded dataset DY11K.
  • Figure 4: The loss of our pre-training (PT), the accuracy (Acc) of development set (Dev) for humor detection (HD) by fine-tuning after PT and without PT by our CVLA.
  • Figure 5: Self-Attention visualization of our multi-modal encoder without and with contrastive pre-training. The four regions separated by gray dashed lines from left to right (top to bottom) represent V, A (video branch), T, C (language branch), respectively.
  • ...and 1 more figures