Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations

TL;DR

The work tackles cross-view generalization for fine-grained video understanding by learning view-invariant representations from unpaired ego- and exocentric videos. It introduces Bootstrap Your Own Views (BYOV), which integrates masked self-view modeling (MSM) to capture temporal dynamics and masked cross-view modeling (MCM) to align representations across views, using selective token merging to focus on action-relevant regions. The model is trained with a joint objective L_BYOV = L_MSM + L_MCM and then deployed with a frozen encoder to produce view-invariant embeddings for downstream tasks. Across the AE2 benchmark and Charades-Ego, BYOV achieves state-of-the-art performance, demonstrating strong cross-view transfer and robustness to viewpoint differences, with practical implications for robotics, AR/VR, and multi-view activity analysis.

Abstract

View-invariant representation learning from egocentric (first-person, ego) and exocentric (third-person, exo) videos is a promising approach toward generalizing video understanding systems across multiple viewpoints. However, this area has been underexplored due to the substantial differences in perspective, motion patterns, and context between ego and exo views. In this paper, we propose a novel masked ego-exo modeling that promotes both causal temporal dynamics and cross-view alignment, called Bootstrap Your Own Views (BYOV), for fine-grained view-invariant video representation learning from unpaired ego-exo videos. We highlight the importance of capturing the compositional nature of human actions as a basis for robust cross-view understanding. Specifically, self-view masking and cross-view masking predictions are designed to learn view-invariant and powerful representations concurrently. Experimental results demonstrate that our BYOV significantly surpasses existing approaches with notable gains across all metrics in four downstream ego-exo video tasks. The code is available at https://github.com/park-jungin/byov.

Figures (7)

• Figure 1: Challenges for view-invariant video representation learning from unpaired ego and exo videos. There is a fundamental gap between ego-exo views, such as perspective differences following the camera angle, context cues, scale and depth variations, and different motion patterns even when doing the activity.
• Figure 2: The overall framework of BYOV. The proposed masked (b) self-view and (c) cross-view modeling encourages learning fine-grained view-invariant representations from unpaired ego and exo videos. We employ an encoder-decoder framework across disparate views, which is trained simultaneously to predict both frame tokens from its own view and frame tokens from a different view performing the same action. As shown in (a), we note that the decoder is discarded in performing downstream tasks.
• Figure 3: Results of frame retrieval from Break Eggs and Pour Liquid. We retrieve the nearest neighbor frames (red box) corresponding to the given query frame (blue box).
• Figure 4: tSNE visualization of frame embeddings (a) before training and (b) trained with BYOV. We sample two ego and two exo videos from Break Eggs.
• Figure A1: Qualitative examples of frame retrieval from the Pour Milk and Tennis Forehand datasets. We retrieve the nearest neighbor frames (red box) corresponding to the given query frame (blue box).