Comparing Traditional and LLM-based Search for Image Geolocation

TL;DR

This study compares traditional keyword-based search with LLM-based conversational search for image geolocation using a six-round GeoGuessr-style task with 60 participants. Traditional search yielded higher accuracy, while LLM-based search produced longer, more natural queries and a tendency to rephrase rather than expand queries. The analysis links performance gaps to differences in query formulation and reformulation strategies, with qualitative data highlighting language barriers and the challenge of effectively communicating intent to LLMs. The findings underscore the importance of interface design and prompt literacy for practical LLM-assisted search and suggest avenues to improve LLM-based information retrieval in real-world tasks.

Abstract

Web search engines have long served as indispensable tools for information retrieval; user behavior and query formulation strategies have been well studied. The introduction of search engines powered by large language models (LLMs) suggested more conversational search and new types of query strategies. In this paper, we compare traditional and LLM-based search for the task of image geolocation, i.e., determining the location where an image was captured. Our work examines user interactions, with a particular focus on query formulation strategies. In our study, 60 participants were assigned either traditional or LLM-based search engines as assistants for geolocation. Participants using traditional search more accurately predicted the location of the image compared to those using the LLM-based search. Distinct strategies emerged between users depending on the type of assistant. Participants using the LLM-based search issued longer, more natural language queries, but had shorter search sessions. When reformulating their search queries, traditional search participants tended to add more terms to their initial queries, whereas participants using the LLM-based search consistently rephrased their initial queries.

Figures (12)

• Figure 1: Initial viewpoints (with the location indicated) of the six rounds in the experiment.
• Figure 2: Performance distribution (points) comparison between Search and LLM conditions on average (left) and per round.
• Figure 3: Percentage of multi-query rounds for Search and LLM conditions
• Figure 4: Comparison of part-of-speech tag counts between Search and LLM conditions
• Figure 5: Average number of query terms for successive queries in a round