Training-free Online Video Step Grounding

TL;DR

The paper tackles online, training-free Video Step Grounding by leveraging zero-shot Large Multimodal Models (LMMs) and introducing BaGLM, a Bayesian grounding framework that fuses LMM predictions with memory from past frames. BaGLM builds a transition model from an LLM-derived step dependency matrix and refines it with action-progress estimates to guide Bayesian updates, while the observation model remains an LMM-based VSG scorer. Across HT-Step, CrossTask, and Ego4D Goal-Step, BaGLM outperforms state-of-the-art offline, training-based methods in online settings, with ablations highlighting the value of readiness/validity priors, LLM choice, and segment duration. The work demonstrates practical impact for real-time AR/VR guidance and remote learning, while acknowledging dependence on LLM quality and the potential for improved priors and generalization in future work.

Abstract

Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.

Figures (7)

• Figure 1: We tackle Video Step Grounding with BaGLM, a training-free approach which combines Bayesian filtering with Large Multimodal Models to enable online inference over video streams.
• Figure 2: Comparison of VSG performance on HT-Step, CrossTask, and Ego4D Goal-Step datasets, prompting LMMs with step options and video segments in an online fashion. For reference, we also show the performance of the top two performing methods from the state of the art (dark bars).
• Figure 3: Overview of BaGLM. Given a sequence of steps, an LLM is used to estimate a dependency matrix among them. This matrix is used to compute step transition probabilities employed during the predict step of a Bayesian filter. As the video progresses, the transition model is updated using estimates of each step's progress from an LMM. The update step of the filter merges this with the predictions from the LMM, refining the output.
• Figure 4: Ablation study on varying the segment duration and on the used LMM.
• Figure 5: R@1 on the HT-Step dataset, grouped by the number of candidate steps in each task.