EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision

TL;DR

EAGLE is presented, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision, and achieves state-ofthe-art performance on the Ego4D-VQ benchmark.

Abstract

Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-ofthe-art performance on the Ego4D-VQ benchmark.

Figures (8)

• Figure 1: Overview of EAGLE. Our framework consists of two branches: VQL-2D and -3D. The VQL-2D features a dual-branch architecture built upon a shared backbone. The segmentation branch serves as a precise identifier, guided by an appearance-aware meta-learning memory (b) to generate pixel-level masks with fine-grained semantic cues. The tracking branch, acting as a navigator, is driven by a geometry-aware localization memory (c) to produce a discriminative score map robust to egocentric view changes. Finally, a decoder fuses the outputs from both branches to yield the results. 3D branch leverages the VGGT to jointly process the 2D results, camera pose, and depth, ultimately predicting the positional offset of the query in 3D space.
• Figure 2: Visualization of qualitative results of VQL-2D. Each row presents the visual query, video frames, the predicted trajectories from both EAGLE, VQLoc, and the ground truth. Additionally, the temporal confidence curve predicted by EAGLE is shown, with the green shaded region indicating the ground-truth interval.
• Figure 3: Visualization of 2D responses and 3D localization. We back-projected the 2D response predictions and query locations into 3D space. Groundtruth, EAGLE, and EgoLoc-V1, were compared. We don't know the size and rotation of the 3D bbox during the prediction. However, for visualization purposes, we utilize the size and rotation from the ground truth annotations and treat the predicted 3D location as the center of the 3D bbox.
• Figure 4: Qualitative analysis on SAM pathways. (a) illustrates segmentation of the largest object based on target pixel proportion using the Everything prompt; (b) demonstrates segmentation centered on the image's center point, based on a center-point prior; (c) shows improved segmentation results by combining a point prompt with a bounding box sized to 2/3 of the query image; (d) reveals that Everything and point prompt can sometimes fail, while the method in (c) exhibits robust performance in most cases.
• Figure 5: Ablation Analysis of Capacity and Storage. The horizontal axis represents the selected memory bank capacity, the left vertical axis corresponds to the range of VQ2D evaluation metrics, and the right vertical axis indicates storage capacity in gigabytes (GB).