Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Transformer for Live Streaming Highlight Prediction

Jiaxin Deng, Shiyao Wang, Dong Shen, Liqin Zhao, Fan Yang, Guorui Zhou, Gaofeng Meng

TL;DR

A novel Border-aware Pairwise Loss is proposed to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal and outperforms various strong baselines on both real-world scenarios and public datasets.

Abstract

Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic.

A Multimodal Transformer for Live Streaming Highlight Prediction

TL;DR

A novel Border-aware Pairwise Loss is proposed to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal and outperforms various strong baselines on both real-world scenarios and public datasets.

Abstract

Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic.
Paper Structure (16 sections, 13 equations, 5 figures, 3 tables)

This paper contains 16 sections, 13 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Highlight Detection
  4. Live Streaming Highlight Detection
  5. Methodology
  6. Problem Formulation and Model Overview
  7. Perceiver Block and Casual Attention
  8. Modality Temporal Alignment Module (MTAM)
  9. Border Aware Pairwise Loss
  10. Experiments
  11. Dataset
  12. Evaluation Metrics
  13. Overall Performance Comparison
  14. Border Aware Pairwise Loss and MTAM
  15. Modality Impact
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The live streaming platform and typical streamers. (a) Cascading UI of live streaming. (b) Highlight moments of two typical streamers.
  • Figure 2: The framework of our method. Part (a) shows the architecture of Perceiver Block and Casual Attention Decoder which are discussed in Sect.\ref{['MultiModalTransformer']}. Part (b) shows the proposed Modality Temporal Alignment Module which is discussed in Sect.\ref{['DTWAlignment']}. Part (c) shows the motivation of Border-aware Pairwise Loss which is discussed in Sect.\ref{['borderAwarePairwiseLoss']}.
  • Figure 3: The change of pairwise loss function w.r.t. $s_i-s_j$ .
  • Figure 4: The change of $L_{Point}$ and $L_{Pair}$ during training.
  • Figure 5: Visualization of Dynamic Time Warping (DTW) alignment results for the KLive dataset.