FlySearch: Exploring how vision-language models explore

TL;DR

FlySearch provides a realistic, outdoor, UAV-enabled benchmark to evaluate Vision-Language Models on object-based exploration in open-world environments. Using Unreal Engine 5 with procedurally generated forest and city scenes, it defines three tasks (FS-1, FS-Anomaly-1, FS-2) to test basic perception, contextual understanding, and long-horizon exploration in a zero-shot setting. The study benchmarking multiple proprietary and open-weight VLMs shows large gaps to human performance, with failure modes including hallucination, mis-grounding, and poor strategic exploration; some improvements arise from fine-tuning but not for the hardest tasks. The work offers a reproducible framework and resources to spur advances in spatial reasoning and exploration in VLMs, with implications for real-world UAV search and safety considerations.

Abstract

The real world is messy and unstructured. Uncovering critical information often requires active, goal-driven exploration. It remains to be seen whether Vision-Language Models (VLMs), which recently emerged as a popular zero-shot tool in many difficult tasks, can operate effectively in such conditions. In this paper, we answer this question by introducing FlySearch, a 3D, outdoor, photorealistic environment for searching and navigating to objects in complex scenes. We define three sets of scenarios with varying difficulty and observe that state-of-the-art VLMs cannot reliably solve even the simplest exploration tasks, with the gap to human performance increasing as the tasks get harder. We identify a set of central causes, ranging from vision hallucination, through context misunderstanding, to task planning failures, and we show that some of them can be addressed by finetuning. We publicly release the benchmark, scenarios, and the underlying codebase.

Figures (12)

• Figure 1: FlySearch is a benchmark that evaluates exploration skills using vision-language reasoning. To complete each assessment scenario, a model must locate an object specified in natural language. The agent controls an Unmanned Aerial Vehicle (UAV) by observing images obtained from successive locations of the UAV and providing text commands describing the next move.
• Figure 2: Evaluation pipeline: FlySearch consists of three parts besides the vision-language model. The simulator renders near-photorealistic views of a large open-world map and handles basic physics such as collisions. The evaluation controller handles the communication between the evaluated vision-language model and the simulator and performs the evaluation. The scenario generator (functionally integrated with the controller and simulator) procedurally generates new evaluation scenarios.
• Figure 3: Environments: Our benchmark consists of two types of evaluation environments, forest and city. For each environment, we can generate an infinite number of procedurally generated test scenarios. The top row shows a preview of the environment, exhibiting the visual fidelity of the simulation. Below are top-down views of objects, matching the perspective of the agent.
• Figure 3: FS-Anomaly-1 results: Success rates ($\pm$ standard errors) of the evaluated models. Full results are in Appendix \ref{['app:additional_exps']}.
• Figure 4: Not claimed successes in FS-1: In this figure we compare the number of cases, where the agent located the object, but failed to claim the FOUND action. We observe, that small models often fail to format the text output to report success.