RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations

TL;DR

The paper addresses Visual Query Localization in long, untrimmed videos by introducing Relocate, a training-free baseline that relies on region-based representations to locate the last appearance of a visual query. It employs a stage-wise pipeline—Prepare, Search, Refine, Track, and Reiterate—building object tokens from region masks, matching them to query tokens via cosine similarity, refining candidates with object-centered crops, and tracking the final occurrence, with query expansion to capture appearance variation. Relocate achieves large improvements on Ego4D VQ2D without task-specific training, underscoring the effectiveness of region-based encodings for efficient, open-world localization and enabling reuse of video encodings across multiple queries. The work also provides extensive ablations and design insights, demonstrating robustness to hyperparameters and highlighting practical considerations like computation time and potential speedups.

Abstract

We present RELOCATE, a simple training-free baseline designed to perform the challenging task of visual query localization in long videos. To eliminate the need for task-specific training and efficiently handle long videos, RELOCATE leverages a region-based representation derived from pretrained vision models. At a high level, it follows the classic object localization approach: (1) identify all objects in each video frame, (2) compare the objects with the given query and select the most similar ones, and (3) perform bidirectional tracking to get a spatio-temporal response. However, we propose some key enhancements to handle small objects, cluttered scenes, partial visibility, and varying appearances. Notably, we refine the selected objects for accurate localization and generate additional visual queries to capture visual variations. We evaluate RELOCATE on the challenging Ego4D Visual Query 2D Localization dataset, establishing a new baseline that outperforms prior task-specific methods by 49% (relative improvement) in spatio-temporal average precision.

Figures (7)

• Figure 1: Relocate is a training-free framework designed for visual query localization. It can effectively localize target objects in long videos despite challenging conditions such as visual clutter, occlusions, background blending, motion blur, viewpoint changes, and brief object appearances. Here we show visual queries on the left and the successfully localized object appearances marked by cyan bounding boxes on the right.
• Figure 2: Relocate framework:Prepare object tokens from the given video $\mathcal{V}$ and search for candidates that match the visual query $\mathcal{Q}$. Then, refine the search results for better precision and track the latest refined candidate across video frames to get a response track prediction $\hat{\mathcal{R}}$. Finally, use the response track to create additional visual queries and reiterate the search, refinement, and tracking process.
• Figure 3: Examples where a wrong object is localized. Among 100 randomly sampled instances from the VQ2D validation set, Relocate localized an instance of the query object in 93 cases, and failed to localize the correct object only under extreme conditions.
• Figure 4: Comparing cosine similarity heatmaps for patch-based and region-based representations. Region-based representations yield distinct, high-similarity matches for query objects (brighter regions), while patch-based representations produce more diffuse similarity patterns.
• Figure 5: Impact of refining search results. Initial candidates are shown in yellow boxes, and refined candidates are shown in green.