AutoTour: Automatic Photo Tour Guide with Smartphones and LLMs

TL;DR

It is demonstrated that AutoTour can deliver rich, interpretable annotations for both iconic and lesser-known landmarks, enabling a new form of interactive, context-aware exploration that bridges visual perception and geospatial understanding.

Abstract

We present AutoTour, a system that enhances user exploration by automatically generating fine-grained landmark annotations and descriptive narratives for photos captured by users. The key idea of AutoTour is to fuse visual features extracted from photos with nearby geospatial features queried from open matching databases. Unlike existing tour applications that rely on pre-defined content or proprietary datasets, AutoTour leverages open and extensible data sources to provide scalable and context-aware photo-based guidance. To achieve this, we design a training-free pipeline that first extracts and filters relevant geospatial features around the user's GPS location. It then detects major landmarks in user photos through VLM-based feature detection and projects them into the horizontal spatial plane. A geometric matching algorithm aligns photo features with corresponding geospatial entities based on their estimated distance and direction. The matched features are subsequently grounded and annotated directly on the original photo, accompanied by large language model-generated textual and audio descriptions to provide an informative, tour-like experience. We demonstrate that AutoTour can deliver rich, interpretable annotations for both iconic and lesser-known landmarks, enabling a new form of interactive, context-aware exploration that bridges visual perception and geospatial understanding.

Figures (16)

• Figure 1: Illustration of a new user interaction approach for touring.
• Figure 2: Comparison of existing models with vision capabilities. The semantic segmentation result is from Segment Anything kirillov2023segany and the vison language model results are from GPT-4o openai2025gpt4o. None of them can achieve the results shown in Fig. \ref{['fig:scenario']}
• Figure 3: Geospatial database comparison between Google Maps Places API and free-source OSM.
• Figure 4: Design workflow of AutoTour.
• Figure 5: Geo-spatial feature extraction pipeline. Data retrieved from OSM is filtered and structured to retain key attributes for subsequent feature matching with photos.