Zero-Shot Temporal Interaction Localization for Egocentric Videos

TL;DR

EgoLoc tackles zero-shot temporal interaction localization (TIL) in egocentric videos by pinpointing the start and end of hand–object contact using a novel combination of 3D hand velocities, self-adaptive sampling, and closed-loop vision–language model reasoning. The method constructs high-quality visual prompts from a grid of nearby frames around low-velocity anchor frames and uses a universal text prompt with a VLM to infer contact and separation timestamps in a single pass, followed by refinement via visual and dynamic cues. The approach outperforms state-of-the-art baselines on EgoPAT3D-DT and the new DeskTIL benchmark, with notable gains in accuracy and efficiency, and shows robust stability across trials. The authors provide open-source code and data to facilitate reproducibility and further development in zero-shot TIL for egocentric HOI tasks.

Abstract

Locating human-object interaction (HOI) actions within video serves as the foundation for multiple downstream tasks, such as human behavior analysis and human-robot skill transfer. Current temporal action localization methods typically rely on annotated action and object categories of interactions for optimization, which leads to domain bias and low deployment efficiency. Although some recent works have achieved zero-shot temporal action localization (ZS-TAL) with large vision-language models (VLMs), their coarse-grained estimations and open-loop pipelines hinder further performance improvements for temporal interaction localization (TIL). To address these issues, we propose a novel zero-shot TIL approach dubbed EgoLoc to locate the timings of grasp actions for human-object interaction in egocentric videos. EgoLoc introduces a self-adaptive sampling strategy to generate reasonable visual prompts for VLM reasoning. By absorbing both 2D and 3D observations, it directly samples high-quality initial guesses around the possible contact/separation timestamps of HOI according to 3D hand velocities, leading to high inference accuracy and efficiency. In addition, EgoLoc generates closed-loop feedback from visual and dynamic cues to further refine the localization results. Comprehensive experiments on the publicly available dataset and our newly proposed benchmark demonstrate that EgoLoc achieves better temporal interaction localization for egocentric videos compared to state-of-the-art baselines. We have released our code and relevant data as open-source at https://github.com/IRMVLab/EgoLoc.

Figures (6)

• Figure I: Compared to the TAL paradigm (a) that localizes a specific action in an untrimmed video with RGB images and action descriptions, TIL aims to find the frames closest to the start and end of a human-object in-contact period. Our proposed EgoLoc paradigm (b) absorbs both 2D and 3D observations and implements accurate temporal interaction localization with a universal text prompt in a closed-loop form.
• Figure II: The overall pipeline of EgoLoc: EgoLoc integrates the self-adaptive sampling strategy, the VLM-based reasoning module, and the closed-loop feedback mechanism. The estimation of the contact timestamp is illustrated as an example for clearer clarification. Firstly, the self-adaptive sampling strategy samples the initial guesses of the contact timestamp using 3D hand velocities from RGB and depth images. Then, the VLM-based reasoning module generates the first-round contact timestamp based on the visual prompt. Finally, the closed-loop feedback mechanism outputs the refined second-round timestamp by assessing hand-object contact states and velocities.
• Figure III: The hand motion extraction module generates 2D hand masks with Grounded SAM ren2024grounded and transforms the hand depth in each camera coordinate system to 3D hand positions in the global coordinate system. The 3D hand positions are further used to calculate 3D hand velocities.
• Figure IV: Visualization of the egocentric HOI data in our newly proposed DeskTIL benchmark.
• Figure V: Visualization of the image frames at the estimated and ground-truth contact/separation timestamps for temporal interaction localization. Here we show the results achieved by each method under their optimal performance conditions in Tab. \ref{['tab:compare_main']}.