Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-grained length controllable video captioning with ordinal embeddings

Tomoya Nitta, Takumi Fukuzawa, Toru Tamaki

TL;DR

This work tackles the problem of fine-grained length and duration control in video captioning by introducing nonlinear multi-hot length embeddings. It develops two concrete embedding schemes—bit embedding and ordinal embedding—alongside a length-level baseline, and demonstrates that ordinal embedding most effectively governs caption length across two diverse datasets. ICA analysis further shows a separation between length signals and semantic content, supporting the idea that length information can be learned without corrupting meaning. The approach also extends to duration control via a lightweight TTS pipeline, highlighting practical utility for temporally constrained narration in real-world applications.

Abstract

This paper proposes a method for video captioning that controls the length of generated captions. Previous work on length control often had few levels for expressing length. In this study, we propose two methods of length embedding for fine-grained length control. A traditional embedding method is linear, using a one-hot vector and an embedding matrix. In this study, we propose methods that represent length in multi-hot vectors. One is bit embedding that expresses length in bit representation, and the other is ordinal embedding that uses the binary representation often used in ordinal regression. These length representations of multi-hot vectors are converted into length embedding by a nonlinear MLP. This method allows for not only the length control of caption sentences but also the control of the time when reading the caption. Experiments using ActivityNet Captions and Spoken Moments in Time show that the proposed method effectively controls the length of the generated captions. Analysis of the embedding vectors with ICA shows that length and semantics were learned separately, demonstrating the effectiveness of the proposed embedding methods.

Fine-grained length controllable video captioning with ordinal embeddings

TL;DR

This work tackles the problem of fine-grained length and duration control in video captioning by introducing nonlinear multi-hot length embeddings. It develops two concrete embedding schemes—bit embedding and ordinal embedding—alongside a length-level baseline, and demonstrates that ordinal embedding most effectively governs caption length across two diverse datasets. ICA analysis further shows a separation between length signals and semantic content, supporting the idea that length information can be learned without corrupting meaning. The approach also extends to duration control via a lightweight TTS pipeline, highlighting practical utility for temporally constrained narration in real-world applications.

Abstract

This paper proposes a method for video captioning that controls the length of generated captions. Previous work on length control often had few levels for expressing length. In this study, we propose two methods of length embedding for fine-grained length control. A traditional embedding method is linear, using a one-hot vector and an embedding matrix. In this study, we propose methods that represent length in multi-hot vectors. One is bit embedding that expresses length in bit representation, and the other is ordinal embedding that uses the binary representation often used in ordinal regression. These length representations of multi-hot vectors are converted into length embedding by a nonlinear MLP. This method allows for not only the length control of caption sentences but also the control of the time when reading the caption. Experiments using ActivityNet Captions and Spoken Moments in Time show that the proposed method effectively controls the length of the generated captions. Analysis of the embedding vectors with ICA shows that length and semantics were learned separately, demonstrating the effectiveness of the proposed embedding methods.
Paper Structure (34 sections, 9 equations, 18 figures, 6 tables)

This paper contains 34 sections, 9 equations, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Autoregressive vs non-autoregressive decoders.
  4. Controlling the probability of EOS.
  5. Prefix learning.
  6. Length embedding.
  7. Length levels.
  8. Use of LLM.
  9. Length embedding
  10. Length Level Embedding
  11. Bit Embedding
  12. Ordinal Embedding
  13. Experiment
  14. Dataset
  15. Experimental Settings
  16. ...and 19 more sections

Figures (18)

  • Figure 1: Histograms of the lengths of annotated captions of \ref{['fig:activitynet_hist']} ActivityNet Captions and \ref{['fig:spoken_mit_hist']} Spoken MiT.
  • Figure 2: Examples of generated captions for a video of ActivityNet Captions with different target lengths and embedding methods.
  • Figure 3: Examples of generated captions for a video of Spoken MiT with different target lengths and embedding methods.
  • Figure 4: Captions generated for the same video in Fig.\ref{['fig:spoken_mit_vid_cap']} with ordinal embedding. The target length ranges between 5 and 20.
  • Figure 5: The difference between the target lengths and the average lengths of generated captions for (a) ActivityNet Captions and (b) Spoken MiT.
  • ...and 13 more figures