Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The MSR-Video to Text Dataset with Clean Annotations

Haoran Chen, Jianmin Li, Simone Frintrop, Xiaolin Hu

TL;DR

This paper identifies pervasive annotation noise in the MSR-VTT video captioning dataset and demonstrates that a structured four-step cleaning pipeline—addressing special characters, spelling mistakes, duplicates, and multi-sentence captions—consistently improves both automated metrics and human judgments. By calibrating duplicate-removal thresholds and sentence handling, the authors show that models trained on the cleaned data (notably VNS-GRU and other baselines) achieve higher BLEU-4, CIDEr, METEOR, and ROUGE-L scores, with gains carried over to cleaned test sets as well. The improvements, quantified through both objective metrics and a human evaluation study, argue for using cleaned annotations as a more reliable ground truth for video captioning research. The cleaned MSR-VTT dataset is advocated for future development and can be extended to other datasets and NLP tasks.

Abstract

Video captioning automatically generates short descriptions of the video content, usually in form of a single sentence. Many methods have been proposed for solving this task. A large dataset called MSR Video to Text (MSR-VTT) is often used as the benchmark dataset for testing the performance of the methods. However, we found that the human annotations, i.e., the descriptions of video contents in the dataset are quite noisy, e.g., there are many duplicate captions and many captions contain grammatical problems. These problems may pose difficulties to video captioning models for learning underlying patterns. We cleaned the MSR-VTT annotations by removing these problems, then tested several typical video captioning models on the cleaned dataset. Experimental results showed that data cleaning boosted the performances of the models measured by popular quantitative metrics. We recruited subjects to evaluate the results of a model trained on the original and cleaned datasets. The human behavior experiment demonstrated that trained on the cleaned dataset, the model generated captions that were more coherent and more relevant to the contents of the video clips.

The MSR-Video to Text Dataset with Clean Annotations

TL;DR

This paper identifies pervasive annotation noise in the MSR-VTT video captioning dataset and demonstrates that a structured four-step cleaning pipeline—addressing special characters, spelling mistakes, duplicates, and multi-sentence captions—consistently improves both automated metrics and human judgments. By calibrating duplicate-removal thresholds and sentence handling, the authors show that models trained on the cleaned data (notably VNS-GRU and other baselines) achieve higher BLEU-4, CIDEr, METEOR, and ROUGE-L scores, with gains carried over to cleaned test sets as well. The improvements, quantified through both objective metrics and a human evaluation study, argue for using cleaned annotations as a more reliable ground truth for video captioning research. The cleaned MSR-VTT dataset is advocated for future development and can be extended to other datasets and NLP tasks.

Abstract

Video captioning automatically generates short descriptions of the video content, usually in form of a single sentence. Many methods have been proposed for solving this task. A large dataset called MSR Video to Text (MSR-VTT) is often used as the benchmark dataset for testing the performance of the methods. However, we found that the human annotations, i.e., the descriptions of video contents in the dataset are quite noisy, e.g., there are many duplicate captions and many captions contain grammatical problems. These problems may pose difficulties to video captioning models for learning underlying patterns. We cleaned the MSR-VTT annotations by removing these problems, then tested several typical video captioning models on the cleaned dataset. Experimental results showed that data cleaning boosted the performances of the models measured by popular quantitative metrics. We recruited subjects to evaluate the results of a model trained on the original and cleaned datasets. The human behavior experiment demonstrated that trained on the cleaned dataset, the model generated captions that were more coherent and more relevant to the contents of the video clips.

Paper Structure

This paper contains 16 sections, 3 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets
  4. Recent Advances in Video Captioning
  5. Analysis and Cleaning of the MSR-VTT dataset
  6. Special Characters
  7. Spelling Mistakes
  8. Duplicate Annotations
  9. Successive Sentences without Punctuations
  10. Experiments
  11. Evaluation Metrics
  12. Influence of Edit Distance Threshold and Similarity Threshold on Duplicates Removal
  13. Comparison between the Original/Cleaned MSR-VTT Datasets
  14. Ablation Study
  15. Human Evaluation
  16. ...and 1 more sections

Figures (6)

  • Figure 1: An example video clip (No. 4290, starting from 0) with duplicate annotations. $\times t$ denotes repeating $t$ times
  • Figure 2: Three examples in the MSR-VTT dataset. The words in blue and red denote grammatical mistakes and spelling mistakes, respectively
  • Figure 3: Redundancy samples in the MSR-VTT dataset. Caption 1 can be divided into three sentences. And Caption 2 can be divided into two or three sentences
  • Figure 4: The performance of typical models on the MSR-VTT dataset during 2016 and 2020. The models include VideoLAB, Aalto, v2t_navigator, MTVC DBLP:conf/acl/PasunuruB17, CIDEnt-RL pasunuru-bansal-2017-reinforced, SibNet liu-sheng-2018-sibnet, HACA wang2018watch, TAMoE wang2019learning, SAM-SS 10.3389/frobt.2020.475767 and POS_RL Wang_2019_ICCV and VNS-GRU chen2020delving. The first three models are from ACM Multimedia MSR-VTT Challenge 2016 acm/2016/Online. VideoLAB was used as the baseline (0% change).
  • Figure 5: An example question in the human evaluation experiment. Captions A and B were generated by VNS-GRU or VNS-GRU*
  • ...and 1 more figures