Robust Ego-Exo Correspondence with Long-Term Memory

TL;DR

This work tackles robust object-level ego-exo correspondence under extreme viewpoint changes and occlusions. It builds LM-EEC, a SAM 2–based framework with a Memory-View Mixture-of-Experts and dual long-term memory compression to fuse ego and exo information for accurate cross-view segmentation. On EgoExo4D, LM-EEC achieves state-of-the-art results, outperforming SAM 2 baselines and prior methods, and exhibits strong generalization across diverse scenarios and object sizes. The approach offers a practical path toward reliable cross-view perception for AR, robotics, and assistive technologies, while highlighting opportunities for modular MV-MoE adaptations and awareness of privacy considerations.

Abstract

Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features (i) a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.

Figures (11)

• Figure 1: Left: Comparison of segmentation results between XView-XMem and our model, using exocentric videos as an example. Right: Quantitative results on the EgoExo4D validation set.
• Figure 2: Overview of our proposed model, which consists of three key components: multi-view encoding, dual memory compression, and object mask prediction. The multi-view encoder extracts features from egocentric and exocentric videos, using a Memory-View MoE module to adaptively combine memory-aware and view-specific representations. The object mask prediction module then generates segmentation masks for objects in the exocentric view. To capture long-term dependencies efficiently, we apply dual memory compression on the memory banks from both viewpoints.
• Figure 2: Ablation on our fusion mechanism.
• Figure 3: Overview of the proposed Memory-View Mixture-of-Experts (MV-MoE) module. Channel- and spatial-wise routers generate dynamic weights to recalibrate memory-aware and view-specific features, enabling adaptive and context-aware fusion of complementary information.
• Figure 4: An illustration of our memory bank compression strategy, which preserves a fixed memory size in both ego-view and exo-view memory banks by aggregating temporally redundant information.