Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An empirical study of the effect of video encoders on Temporal Video Grounding

Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Felipe Bravo-Marquez

TL;DR

This work systematically investigates how video encoder choices affect Temporal Video Grounding by keeping a classic TVG backbone fixed and evaluating a wide range of pre-trained video representations across Charades-STA, ActivityNet-Captions, and YouCookII. The study uncovers substantial performance variability driven solely by the video features, with clear patterns where Transformer- and CNN-based encoders excel on different datasets and evidence of feature complementarity. It also demonstrates orthogonality in predictive biases across features, suggesting that combining diverse encoders could yield further gains beyond any single feature set. Overall, the results challenge the notion that architectural complexity alone drives TVG progress and highlight the value of feature diversity and principled pretraining for robust temporal localization.

Abstract

Temporal video grounding is a fundamental task in computer vision, aiming to localize a natural language query in a long, untrimmed video. It has a key role in the scientific community, in part due to the large amount of video generated every day. Although we find extensive work in this task, we note that research remains focused on a small selection of video representations, which may lead to architectural overfitting in the long run. To address this issue, we propose an empirical study to investigate the impact of different video features on a classical architecture. We extract features for three well-known benchmarks, Charades-STA, ActivityNet-Captions and YouCookII, using video encoders based on CNNs, temporal reasoning and transformers. Our results show significant differences in the performance of our model by simply changing the video encoder, while also revealing clear patterns and errors derived from the use of certain features, ultimately indicating potential feature complementarity.

An empirical study of the effect of video encoders on Temporal Video Grounding

TL;DR

This work systematically investigates how video encoder choices affect Temporal Video Grounding by keeping a classic TVG backbone fixed and evaluating a wide range of pre-trained video representations across Charades-STA, ActivityNet-Captions, and YouCookII. The study uncovers substantial performance variability driven solely by the video features, with clear patterns where Transformer- and CNN-based encoders excel on different datasets and evidence of feature complementarity. It also demonstrates orthogonality in predictive biases across features, suggesting that combining diverse encoders could yield further gains beyond any single feature set. Overall, the results challenge the notion that architectural complexity alone drives TVG progress and highlight the value of feature diversity and principled pretraining for robust temporal localization.

Abstract

Temporal video grounding is a fundamental task in computer vision, aiming to localize a natural language query in a long, untrimmed video. It has a key role in the scientific community, in part due to the large amount of video generated every day. Although we find extensive work in this task, we note that research remains focused on a small selection of video representations, which may lead to architectural overfitting in the long run. To address this issue, we propose an empirical study to investigate the impact of different video features on a classical architecture. We extract features for three well-known benchmarks, Charades-STA, ActivityNet-Captions and YouCookII, using video encoders based on CNNs, temporal reasoning and transformers. Our results show significant differences in the performance of our model by simply changing the video encoder, while also revealing clear patterns and errors derived from the use of certain features, ultimately indicating potential feature complementarity.
Paper Structure (13 sections, 2 figures, 1 table)

This paper contains 13 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Action Classification
  4. Temporal Video Grounding
  5. Experimental Framework
  6. Model
  7. Video Representation
  8. Datasets
  9. Evaluation
  10. Results
  11. Quantitative Analysis
  12. Qualitative Analysis
  13. Conclusions

Figures (2)

  • Figure 1: The distribution of normalized temporal predictions with extracted features. The x-axis represents the prediction start time, while the y-axis represents the prediction end time. The graph shows normalized predictions for the Charades dataset (top) and the ActivityNet dataset (bottom), using features from MViT (left), I3D NLN (center) and TSM (right).
  • Figure 2: Overlap of the correct predictions, in terms of query-video pairs for $\alpha=0.7$, in Charades-STA (left), ActivityNet (center) and YoucookII (right) benchmarks. We group features based on their nature.