Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dense Video Captioning Using Unsupervised Semantic Information

Valter Estevam, Rayson Laroca, Helio Pedrini, David Menotti

TL;DR

This work tackles dense video captioning using visual information alone by introducing an unsupervised semantic descriptor learned from short-clip co-occurrences. A visual codebook is created via mini-batch $k$-means, and a co-occurrence probability model inspired by GloVe yields dense semantic vectors used to augment visual features. The descriptor is integrated into a bi-modal transformer for event proposal and a vanilla transformer for captioning, achieving state-of-the-art results among visual-only methods and competitive performance with multi-modal baselines on ActivityNet Captions. The approach reduces reliance on audio/speech modalities while maintaining strong captioning quality, and it opens avenues for unsupervised, cross-video semantic learning in dense video understanding.

Abstract

We introduce a method to learn unsupervised semantic visual information based on the premise that complex events can be decomposed into simpler events and that these simple events are shared across several complex events. We first employ a clustering method to group representations producing a visual codebook. Then, we learn a dense representation by encoding the co-occurrence probability matrix for the codebook entries. This representation leverages the performance of the dense video captioning task in a scenario with only visual features. For example, we replace the audio signal in the BMT method and produce temporal proposals with comparable performance. Furthermore, we concatenate the visual representation with our descriptor in a vanilla transformer method to achieve state-of-the-art performance in the captioning subtask compared to the methods that explore only visual features, as well as a competitive performance with multi-modal methods. Our code is available at https://github.com/valterlej/dvcusi.

Dense Video Captioning Using Unsupervised Semantic Information

TL;DR

This work tackles dense video captioning using visual information alone by introducing an unsupervised semantic descriptor learned from short-clip co-occurrences. A visual codebook is created via mini-batch -means, and a co-occurrence probability model inspired by GloVe yields dense semantic vectors used to augment visual features. The descriptor is integrated into a bi-modal transformer for event proposal and a vanilla transformer for captioning, achieving state-of-the-art results among visual-only methods and competitive performance with multi-modal baselines on ActivityNet Captions. The approach reduces reliance on audio/speech modalities while maintaining strong captioning quality, and it opens avenues for unsupervised, cross-video semantic learning in dense video understanding.

Abstract

We introduce a method to learn unsupervised semantic visual information based on the premise that complex events can be decomposed into simpler events and that these simple events are shared across several complex events. We first employ a clustering method to group representations producing a visual codebook. Then, we learn a dense representation by encoding the co-occurrence probability matrix for the codebook entries. This representation leverages the performance of the dense video captioning task in a scenario with only visual features. For example, we replace the audio signal in the BMT method and produce temporal proposals with comparable performance. Furthermore, we concatenate the visual representation with our descriptor in a vanilla transformer method to achieve state-of-the-art performance in the captioning subtask compared to the methods that explore only visual features, as well as a competitive performance with multi-modal methods. Our code is available at https://github.com/valterlej/dvcusi.
Paper Structure (19 sections, 16 equations, 4 figures, 4 tables)

This paper contains 19 sections, 16 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Event Proposal Generation
  4. Video Captioning
  5. Unsupervised Representation Learning
  6. Methodology
  7. Co-Occurrence Similarity Estimation Module
  8. Video Captioning Module
  9. Event Proposal Module
  10. Training Procedure
  11. Dataset and Evaluation Metrics
  12. Results
  13. Results for Event Proposals
  14. Results for the Video Captioning Stage
  15. Qualitative Analysis
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Examples of visual similarities. (a) Two video fragments with about 28 seconds from YouTube (vdBNZf90PLJ0 and vj3QSVhAhDc). They share some visual similar short clips. (b) A 2D t-SNE representation for the whole visual vocabulary. Some shared fragments are highlighted in red.
  • Figure 2: Overview of the proposed method. (a) describes the event proposal phase, which consists of two stages. In the first stage, a bi-modal transformer is employed in a captioning task, where visual and semantic co-occurrence-based features are used to learn the encoder parameters conditioned by language. Then, in the second stage, these encoder parameters are used to predict temporal event proposals. In the second stage (b), these proposals are used to generate captions using a vanilla transformer and a language generator trained with ground-truth events and sentences.
  • Figure 3: Qualitative comparison on the results including our semantic descriptor in the mdvc method. We show results for event proposals captioning considering the highest Intersection over Union for each ground truth event proposal in the following videos (a) svWiQtzgtOc, (b) P4PQ5tC3gX8, (c) BhAQhPasmhU, and (d) 045Tkq12H_c.
  • Figure 4: Captioning performance levels for ground truth and event proposals, (a) and (b), and for proposal generation (c) considering the model replacing the audio signal with several semantic descriptors generated with different vocabularies (100, 500, 1000, 1500, 2000) and context windows (10s, 30s, 60s).