Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting

TL;DR

This work tackles automatic live video commenting (ALVC) by addressing the need for sentiment-diverse, multi-modal outputs. It introduces So-TVAE, a sentiment-oriented Transformer-based variational autoencoder that combines a sentiment-guided Gaussian-mixture latent space with a sentiment-oriented random mask and a batch attention module to produce diverse, sentiment-controlled comments. The approach leverages a multi-modal encoder with co-attention, a sentiment-prediction module, and a Transformer decoder, optimized with an ELBO-based objective plus a sentiment loss, and demonstrates substantial gains over state-of-the-art methods on Livebot and VideoIC, with further generalization to image-news commenting. The results show improved quality and diversity, and the framework offers controllability for targeted sentiment generation, addressing data-imbalance challenges and advancing interactive, sentiment-aware ALVC capabilities.

Abstract

Automatic live video commenting is with increasing attention due to its significance in narration generation, topic explanation, etc. However, the diverse sentiment consideration of the generated comments is missing from the current methods. Sentimental factors are critical in interactive commenting, and lack of research so far. Thus, in this paper, we propose a Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network which consists of a sentiment-oriented diversity encoder module and a batch attention module, to achieve diverse video commenting with multiple sentiments and multiple semantics. Specifically, our sentiment-oriented diversity encoder elegantly combines VAE and random mask mechanism to achieve semantic diversity under sentiment guidance, which is then fused with cross-modal features to generate live video comments. Furthermore, a batch attention module is also proposed in this paper to alleviate the problem of missing sentimental samples, caused by the data imbalance, which is common in live videos as the popularity of videos varies. Extensive experiments on Livebot and VideoIC datasets demonstrate that the proposed So-TVAE outperforms the state-of-the-art methods in terms of the quality and diversity of generated comments. Related code is available at https://github.com/fufy1024/So-TVAE.

Figures (11)

• Figure 1: A live video commenting example in Livebot with selected video frames and live comments. Green: Positive comments. Black: neutral comments. Blue: Negative comments.
• Figure 2: Overview of our proposed Sentiment-oriented Transformer-based Variational Autoencoder (So-TVAE) network. Firstly, the multi-modal encoder and sentiment-oriented diversity encoder (Part A) are used to extract the context features and the diverse sentimental features respectively, then all features are entered into the comment decoder for generation, which is the backbone of So-TVAE. Moreover, a batch attention module (Part B) is used to extract the batch-level context features and paired with the sentimental features to input into a parameter-shared decoder for generation during the training stage. Besides, the sentiment prediction module is used for single comment generation at inference.
• Figure 2: Human evaluation result on the Livebot test split.
• Figure 3: Quantity distribution curve of meaningful generated comments.
• Figure 4: Visualization of generated comments representation in 2D space with using t-SNE. We plot the scatter diagrams for $8,000$ samples.