Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViLCo-Bench: VIdeo Language COntinual learning Benchmark

Tianqi Tang, Shohreh Deldari, Hao Xue, Celso De Melo, Flora D. Salim

TL;DR

ViLCo-Bench delivers the first dedicated video-language continual learning benchmark and a memory-efficient model to tackle long, multimodal streams. By defining three tasks (MQ, NLQ, VQ) and curating Ego4D-derived data, the work establishes standardized evaluation for cross-modal, temporally-aware continual learning and introduces a dual-memory plus self-supervised framework to mitigate memory and misalignment challenges. Empirical results show ViLCo surpasses traditional CL baselines in both recall and robustness to forgetting, highlighting the value of episodic memory and narration-based SSL for video-language tasks. The benchmark and results lay groundwork for advancing multimodal continual learning beyond classification, with practical implications for embodied AI and interactive video understanding, while acknowledging limitations and avenues for future expansion.

Abstract

Video language continual learning involves continuously adapting to information from video and text inputs, enhancing a model's ability to handle new tasks while retaining prior knowledge. This field is a relatively under-explored area, and establishing appropriate datasets is crucial for facilitating communication and research in this field. In this study, we present the first dedicated benchmark, ViLCo-Bench, designed to evaluate continual learning models across a range of video-text tasks. The dataset comprises ten-minute-long videos and corresponding language queries collected from publicly available datasets. Additionally, we introduce a novel memory-efficient framework that incorporates self-supervised learning and mimics long-term and short-term memory effects. This framework addresses challenges including memory complexity from long video clips, natural language complexity from open queries, and text-video misalignment. We posit that ViLCo-Bench, with greater complexity compared to existing continual learning benchmarks, would serve as a critical tool for exploring the video-language domain, extending beyond conventional class-incremental tasks, and addressing complex and limited annotation issues. The curated data, evaluations, and our novel method are available at https://github.com/cruiseresearchgroup/ViLCo.

ViLCo-Bench: VIdeo Language COntinual learning Benchmark

TL;DR

ViLCo-Bench delivers the first dedicated video-language continual learning benchmark and a memory-efficient model to tackle long, multimodal streams. By defining three tasks (MQ, NLQ, VQ) and curating Ego4D-derived data, the work establishes standardized evaluation for cross-modal, temporally-aware continual learning and introduces a dual-memory plus self-supervised framework to mitigate memory and misalignment challenges. Empirical results show ViLCo surpasses traditional CL baselines in both recall and robustness to forgetting, highlighting the value of episodic memory and narration-based SSL for video-language tasks. The benchmark and results lay groundwork for advancing multimodal continual learning beyond classification, with practical implications for embodied AI and interactive video understanding, while acknowledging limitations and avenues for future expansion.

Abstract

Video language continual learning involves continuously adapting to information from video and text inputs, enhancing a model's ability to handle new tasks while retaining prior knowledge. This field is a relatively under-explored area, and establishing appropriate datasets is crucial for facilitating communication and research in this field. In this study, we present the first dedicated benchmark, ViLCo-Bench, designed to evaluate continual learning models across a range of video-text tasks. The dataset comprises ten-minute-long videos and corresponding language queries collected from publicly available datasets. Additionally, we introduce a novel memory-efficient framework that incorporates self-supervised learning and mimics long-term and short-term memory effects. This framework addresses challenges including memory complexity from long video clips, natural language complexity from open queries, and text-video misalignment. We posit that ViLCo-Bench, with greater complexity compared to existing continual learning benchmarks, would serve as a critical tool for exploring the video-language domain, extending beyond conventional class-incremental tasks, and addressing complex and limited annotation issues. The curated data, evaluations, and our novel method are available at https://github.com/cruiseresearchgroup/ViLCo.
Paper Structure (16 sections, 3 equations, 5 figures, 9 tables)

This paper contains 16 sections, 3 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Backgrounds and Related Works
  3. ViLCo-Bench: Video-Language Continual Learning Benchmark
  4. Unique Video-Language Continual Learning Tasks
  5. Dataset Curation
  6. ViLCo Model
  7. Evaluation Metrics
  8. Evaluation and Discussion
  9. Query Incremental challenge
  10. Ablation Study
  11. Discussion and Limitations
  12. Related Works on Continual Learning
  13. NLQ Query Templates
  14. SSL with Narations
  15. Training Details
  16. ...and 1 more sections

Figures (5)

  • Figure 1: ViLCo-Bench benchmark pipeline.
  • Figure 2: Comparisons with existing continual learning benchmarks.
  • Figure 3: Illustration of the proposed query-incremental challenges.
  • Figure 4: Overview of ViLCo-Bench employing episodic memory modules. All the modules except task-specific headers are shared among tasks and continually get updated.
  • Figure 5: Impact of the order of tasks in Moment Query scenario: BwF (left) and Average Recall (Avg. R@1) (right)