Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Models

TL;DR

MilSCORE addresses the lack of realistic long-context benchmarks for geospatial military decision-making. It introduces a scenario-level benchmark with expert-authored, multi-hop questions grounded in training OPORDs and COA maps, spanning 7 spatial categories and 3 difficulty tiers. The authors propose an evaluation protocol using a tool-using chain-of-thought agent and provide baseline results across contemporary vision-language models, showing substantial headroom and failure modes in long-context planning. The work positions MilSCORE as a challenging testbed to drive improvements in spatially grounded, high-stakes reasoning for LLMs/VLMs.

Abstract

As large language models (LLMs) are applied to increasingly longer and more complex tasks, there is a growing need for realistic long-context benchmarks that require selective reading and integration of heterogeneous, multi-modal information sources. This need is especially acute for geospatial planning problems, such as those found in planning for large-scale military operations, which demand fast and accurate reasoning over maps, orders, intelligence reports, and other distributed data. To address this gap, we present MilSCORE (Military Scenario Contextual Reasoning), to our knowledge the first scenario-level dataset of expert-authored, multi-hop questions grounded in a complex, simulated military planning scenario used for training. MilSCORE is designed to evaluate high-stakes decision-making and planning, probing LLMs' ability to combine tactical and spatial reasoning across multiple sources and to reason over long-horizon, geospatially rich context. The benchmark includes a diverse set of question types across seven categories targeting both factual recall and multi-step reasoning about constraints, strategy, and spatial analysis. We provide an evaluation protocol and report baseline results for a range of contemporary vision-language models. Our findings highlight substantial headroom on MilSCORE, indicating that current systems struggle with realistic, scenario-level long-context planning, and positioning MilSCORE as a challenging testbed for future work.

Figures (2)

• Figure 1: Example MilSCORE questions grouped by spatial-analysis content categories. The figure illustrates representative items from each category: Understanding where; Measuring size, shape, and distribution; Determining how places are related; Finding optimal locations and paths; Detecting and quantifying patterns; Making predictions; and Unsolvable Tasks---including both multiple-choice and free-text formats, reflecting the diversity and complexity of real-world military geospatial reasoning scenarios.
• Figure 2: Overview of the MilSCORE dataset and evaluation pipeline. Unclassified scenario folders containing readable maps, operation texts, and optional structured GeoJSON are used by human experts to author 50 military decision-making tasks spanning single-hop and multi-hop reasoning tiers. For evaluation, a tool-using chain-of-thought agent iteratively inspects the maps and documents, reasons about the question, and produces a free-text or structured answer (optionally with a visualization). An LLM-based grader then compares the final answer against the expert reference to assign a discrete correctness score.