Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompting Visual-Language Models for Efficient Video Understanding

Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, Weidi Xie

TL;DR

This work demonstrates that image-based visual-language models can be efficiently extended to video understanding by learning lightweight continuous prompt vectors and a compact temporal Transformer, while keeping the pre-trained encoders fixed. Framed as a unified, task-agnostic adaptation, the approach supports action recognition, action localisation, and text-video retrieval under closed-set, few-shot, and zero-shot regimes. Through extensive ablations on 10 public datasets, the method shows that prompt learning and temporal modeling yield consistent gains, attaining competitive or state-of-the-art performance with orders of magnitude fewer trainable parameters. The results highlight the practical impact of parameter-efficient, open-vocabulary video understanding and point to future directions in motion representation and prompt semantics.

Abstract

Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for zero-shot generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as continuous prompt vectors, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On 10 public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.

Prompting Visual-Language Models for Efficient Video Understanding

TL;DR

This work demonstrates that image-based visual-language models can be efficiently extended to video understanding by learning lightweight continuous prompt vectors and a compact temporal Transformer, while keeping the pre-trained encoders fixed. Framed as a unified, task-agnostic adaptation, the approach supports action recognition, action localisation, and text-video retrieval under closed-set, few-shot, and zero-shot regimes. Through extensive ablations on 10 public datasets, the method shows that prompt learning and temporal modeling yield consistent gains, attaining competitive or state-of-the-art performance with orders of magnitude fewer trainable parameters. The results highlight the practical impact of parameter-efficient, open-vocabulary video understanding and point to future directions in motion representation and prompt semantics.

Abstract

Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for zero-shot generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as continuous prompt vectors, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On 10 public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.
Paper Structure (39 sections, 5 equations, 1 figure, 11 tables)

This paper contains 39 sections, 5 equations, 1 figure, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Image-based Visual-Language Model
  5. Prompting CLIP for Video Understanding
  6. Problem Scenario.
  7. Model Adaptation by Learning Prompts.
  8. Temporal Modeling.
  9. Training Loss.
  10. Experiments
  11. Implementation Details
  12. Action Recognition
  13. Closed-set Action Recognition
  14. Few-shot Action Recognition
  15. Zero-shot Action Recognition
  16. ...and 24 more sections

Figures (1)

  • Figure 1: Framework Overview. We prepend / append several learnable prompt vectors to the CLIP text encoder to generate action classifiers or query embeddings; adopt a lightweight Transformer on the top of the CLIP image encoder for temporal modeling. During training, both the image and text encoders are kept frozen. By optimising the task-specific prompt vectors and temporal Transformer, we efficiently adapt CLIP to various video understanding tasks: action recognition, text-video retrieval, and action localisation, across closed-set, few-shot, and zero-shot scenarios.