Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties

TL;DR

This work implements Eilev, a novel training paradigm that induces in-context learning over video and text by capturing key properties of pre-training data found by prior work to be essential for in-context learning in transformers.

Abstract

A major reason behind the recent success of large language models (LLMs) is their \textit{in-context learning} capability, which makes it possible to rapidly adapt them to downstream text-based tasks by prompting them with a small number of relevant demonstrations. While large vision-language models (VLMs) have recently been developed for tasks requiring both text and images, they largely lack in-context learning over visual information, especially in understanding and generating text about videos. In this work, we implement \textbf{E}mergent \textbf{I}n-context \textbf{Le}arning on \textbf{V}ideos (\eilev{}), a novel training paradigm that induces in-context learning over video and text by capturing key properties of pre-training data found by prior work to be essential for in-context learning in transformers. In our experiments, we show that \eilev-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. Furthermore, we demonstrate that these key properties of bursty distributions, skewed marginal distributions, and dynamic meaning each contribute to varying degrees to VLMs' in-context learning capability in narrating procedural videos. Our results, analysis, and \eilev{}-trained models yield numerous insights about the emergence of in-context learning over video and text, creating a foundation for future work to optimize and scale VLMs for open-domain video understanding and reasoning. Our code and demo are available at \url{https://github.com/yukw777/EILEV}.

Figures (12)

• Figure 1: In our proposed training procedure $\textbf{EILeV}$, we ensure that the training data satisfy the following three properties: (a) bursty distributions, (b) skewed marginal distributions, and (c) dynamic meanings. Then, we ablate each property to demonstrate its importance. We ablate property (a) by randomly sampling in-context examples; we ablate property (b) by varying the number of common actions in the training data; we ablate property (c) by canonicalizing verbs and nouns using their corresponding verb and noun classes.
• Figure 2: Performance of $\textbf{EILeV}$-trained and off-the-shelf VLMs (Kosmos-2 and Otter) on the evaluation set of held-out rare actions from Ego4D.
• Figure 3: t-SNE plots of the video embeddings from the frozen vision encoder of BLIP-2 OPT-2.7B. Ego4D videos are in red, and EPIC-KITCHENS-100 videos are in blue. Plots for a randomly sampled subset of 40k videos from both and three most common actions from EPIC-KITCHENS-100 are shown. We manually map Ego4D actions to the EPIC-KITCHENS-100 actions.
• Figure 4: Performance of $\textbf{EILeV}$-trained and off-the-shelf VLMs (Kosmos-2 and Otter) on the validation set of out-of-distribution actions from EPIC-KITCHENS-100.
• Figure 5: Results for the bursty distributions ablation experiment.