Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization

TL;DR

NLCO introduces a large-scale, end-to-end benchmark for reasoning over natural-language combinatorial optimization problems. It formalizes a four-layer taxonomy (VarSort, Family, Global, ObjClass) and provides 43 CO tasks across three difficulty levels, with contextualized natural-language instances and solver-annotated references. Through extensive experiments on open-weight and proprietary LLMs, the study reveals strong feasibility and near-optimality on small instances, but rapid degradation with size, especially for graph-structured and bottleneck problems, and highlights the substantial role of reasoning cost and token budgets. The findings underscore the need for more efficient, globally coherent reasoning and for benchmark resources like NLCO to diagnose and guide future improvements in language-driven optimization.

Abstract

While large language models (LLMs) have shown strong performance in math and logic reasoning, their ability to handle combinatorial optimization (CO) -- searching high-dimensional solution spaces under hard constraints -- remains underexplored. To bridge the gap, we introduce NLCO, a \textbf{N}atural \textbf{L}anguage \textbf{C}ombinatorial \textbf{O}ptimization benchmark that evaluates LLMs on end-to-end CO reasoning: given a language-described decision-making scenario, the model must output a discrete solution without writing code or calling external solvers. NLCO covers 43 CO problems and is organized using a four-layer taxonomy of variable types, constraint families, global patterns, and objective classes, enabling fine-grained evaluation. We provide solver-annotated solutions and comprehensively evaluate LLMs by feasibility, solution optimality, and reasoning efficiency. Experiments across a wide range of modern LLMs show that high-performing models achieve strong feasibility and solution quality on small instances, but both degrade as instance size grows, even if more tokens are used for reasoning. We also observe systematic effects across the taxonomy: set-based tasks are relatively easy, whereas graph-structured problems and bottleneck objectives lead to more frequent failures.

Figures (7)

• Figure 1: Overview of the NLCO benchmark.
• Figure 2: Aggregated performance across (a) $\text{VarSort}$ and (b) $\text{ObjClass}$ dimensions in NLCO taxonomy.
• Figure 3: AFR–ALOG joint distribution by family (left) and infeasibility mode distribution (right). FormatError is raised when the solution cannot be parsed with missing fields or non-finite numbers. Unsolvable denotes instances that remain unsolved under the token limit.
• Figure 4: Data profiles across all difficulty tiers. (a) AFR vs. Token budget; (b) Acc. vs. Token budget.
• Figure 7: LLM performance profiles on different NLCO difficulty tiers. (a) Set-S; (b) Set-M; (c) Set-L.