LocalBench: Benchmarking LLMs on County-Level Local Knowledge and Reasoning

TL;DR

LocalBench introduces the first county-level locality benchmark to probe hyper-local knowledge and reasoning in LLMs, grounded in the Localness Conceptual Framework. By integrating census indicators, local discourse, and regional news across 526 counties, it enables a comprehensive assessment of physical, cognitive, and relational locality under closed-book and web-augmented conditions for 13 models. The findings reveal persistent gaps in narrative and especially numerical local reasoning, with web augmentation and scaling showing model-dependent or even negative effects, underscoring architectural mismatches and the need for place-aware, geographically grounded AI. The work highlights practical implications for equitable, place-aware AI and motivates future research on explicit geographic reasoning and community-centered model design.

Abstract

Large language models (LLMs) have been widely evaluated on macro-scale geographic tasks, such as global factual recall, event summarization, and regional reasoning. Yet, their ability to handle hyper-local knowledge remains poorly understood. This gap is increasingly consequential as real-world applications, from civic platforms to community journalism, demand AI systems that can reason about neighborhood-specific dynamics, cultural narratives, and local governance. Existing benchmarks fall short in capturing this complexity, often relying on coarse-grained data or isolated references. We present LocalBench, the first benchmark designed to systematically evaluate LLMs on county-level local knowledge across the United States. Grounded in the Localness Conceptual Framework, LocalBench includes 14,782 validated question-answer pairs across 526 U.S. counties in 49 states, integrating diverse sources such as Census statistics, local subreddit discourse, and regional news. It spans physical, cognitive, and relational dimensions of locality. Using LocalBench, we evaluate 13 state-of-the-art LLMs under both closed-book and web-augmented settings. Our findings reveal critical limitations: even the best-performing models reach only 56.8% accuracy on narrative-style questions and perform below 15.5% on numerical reasoning. Moreover, larger model size and web augmentation do not guarantee better performance, for example, search improves Gemini's accuracy by +13.6%, but reduces GPT-series performance by -11.4%. These results underscore the urgent need for language models that can support equitable, place-aware AI systems: capable of engaging with the diverse, fine-grained realities of local communities across geographic and cultural contexts.

Figures (1)

• Figure 1: LocalBench construction pipeline. The process involves QA generation using a reasoning model and quality analysis via LLM-based assessment and filtering.