Towards Autonomous UAV Visual Object Search in City Space: Benchmark and Agentic Methodology

TL;DR

This work introduces CityAVOS, the first benchmark for autonomous visual object search (AVOS) in city environments, and PRPSearcher, an MLLM-powered agent that mirrors human perception, reasoning, and planning through three maps (semantic, cognitive, uncertainty) and an Inspiration Promote Thought prompting mechanism to balance exploration and exploitation. PRPSearcher leverages an object-centric 3D dynamic semantic map for efficient spatial perception, a 3D cognitive map using attraction values for target reasoning, and a 3D uncertainty map to guide exploration; an IPT-based planning module fuses exploitation and exploration with a threshold-driven prompt strategy. Experiments on CityAVOS show significant improvements over baselines in success rate and search efficiency, though a gap remains with human performance, underscoring the need for stronger semantic reasoning and spatial exploration in urban AVOS. The work provides a solid foundation for future embodied target search in city spaces and offers publicly available dataset and code to spur further research.

Abstract

Aerial Visual Object Search (AVOS) tasks in urban environments require Unmanned Aerial Vehicles (UAVs) to autonomously search for and identify target objects using visual and textual cues without external guidance. Existing approaches struggle in complex urban environments due to redundant semantic processing, similar object distinction, and the exploration-exploitation dilemma. To bridge this gap and support the AVOS task, we introduce CityAVOS, the first benchmark dataset for autonomous search of common urban objects. This dataset comprises 2,420 tasks across six object categories with varying difficulty levels, enabling comprehensive evaluation of UAV agents' search capabilities. To solve the AVOS tasks, we also propose PRPSearcher (Perception-Reasoning-Planning Searcher), a novel agentic method powered by multi-modal large language models (MLLMs) that mimics human three-tier cognition. Specifically, PRPSearcher constructs three specialized maps: an object-centric dynamic semantic map enhancing spatial perception, a 3D cognitive map based on semantic attraction values for target reasoning, and a 3D uncertainty map for balanced exploration-exploitation search. Also, our approach incorporates a denoising mechanism to mitigate interference from similar objects and utilizes an Inspiration Promote Thought (IPT) prompting mechanism for adaptive action planning. Experimental results on CityAVOS demonstrate that PRPSearcher surpasses existing baselines in both success rate and search efficiency (on average: +37.69% SR, +28.96% SPL, -30.69% MSS, and -46.40% NE). While promising, the performance gap compared to humans highlights the need for better semantic reasoning and spatial exploration capabilities in AVOS tasks. This work establishes a foundation for future advances in embodied target search. Dataset and source code are available at https://anonymous.4open.science/r/CityAVOS-3DF8.

Figures (10)

• Figure 1: Examples of six object categories and dataset statistics of the CityAVOS.
• Figure 2: Overview of the agentic method--PRPSearcher.
• Figure 3: Two selected cases of PRPSearcher on two episodes. One is a successful case, demonstrating how and why our proposed approach effectively finds and identifies the target compared to baseline models. While another one is a failed case, highlighting the limitations of our approach in visual reasoning capability compared to human agents.
• Figure 4: The task in the CityAVOS dataset.
• Figure 5: Prompt for related semantics.