Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild

TL;DR

This work addresses outdoor visual search under imperfect priors by introducing Search-TTA, a modular test-time adaptation framework that aligns a satellite image encoder with CLIP space and online refines target priors using onboard measurements inspired by Spatial Poisson Point Processes. The AVS-Bench dataset provides a large-scale, taxonomy-rich benchmark for evaluating multimodal priors in outdoor search and supports zero-shot cross-modal generalization to text and sound. Empirical results show significant gains over planners and baselines, with competitive performance against larger VLMs and successful hardware validation on UAV-like setups. Limitations include sensor simplifications and single-target focus, pointing to future work on multi-target continual learning and richer modalities.

Abstract

To perform outdoor visual navigation and search, a robot may leverage satellite imagery to generate visual priors. This can help inform high-level search strategies, even when such images lack sufficient resolution for target recognition. However, many existing informative path planning or search-based approaches either assume no prior information, or use priors without accounting for how they were obtained. Recent work instead utilizes large Vision Language Models (VLMs) for generalizable priors, but their outputs can be inaccurate due to hallucination, leading to inefficient search. To address these challenges, we introduce Search-TTA, a multimodal test-time adaptation framework with a flexible plug-and-play interface compatible with various input modalities (e.g., image, text, sound) and planning methods (e.g., RL-based). First, we pretrain a satellite image encoder to align with CLIP's visual encoder to output probability distributions of target presence used for visual search. Second, our TTA framework dynamically refines CLIP's predictions during search using uncertainty-weighted gradient updates inspired by Spatial Poisson Point Processes. To train and evaluate Search-TTA, we curate AVS-Bench, a visual search dataset based on internet-scale ecological data containing 380k images and taxonomy data. We find that Search-TTA improves planner performance by up to 30.0%, particularly in cases with poor initial CLIP predictions due to domain mismatch and limited training data. It also performs comparably with significantly larger VLMs, and achieves zero-shot generalization via emergent alignment to unseen modalities. Finally, we deploy Search-TTA on a real UAV via hardware-in-the-loop testing, by simulating its operation within a large-scale simulation that provides onboard sensing.

Figures (17)

• Figure 1: Visual search for bears by a simulated UAV over Yosemite Valley (2). (3) Utilizing a poor probability map leads to suboptimal search performance. (4) Test-time adaptation refines the target probability map via regularized gradient updates by incorporating onboard measurements collected during search, guiding the UAV toward denser and less populated vegetation where bears are more likely to be found usgs2022naipinat2022inat.
• Figure 2: Visual search dataset The taxonomic targets van2021inat cannot be directly seen on these satellite image van2016sentinel, thus prompting the need to rely on indirect visual cues to achieve efficient search.
• Figure 2: Comparing vision models (Found %) $\uparrow$
• Figure 3: Search-TTA Framework. During inference, given a satellite image and query inputs of other modalities, a score map is generated to guide the planner towards areas of high probability. The score map (initially poor) will be updated via SPPP gradient updates to the satellite image encoder during the search process.
• Figure 4: VLM inference time