ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding

TL;DR

CONSINT-Bench introduces a dynamic, live benchmark for real-world consumer intent understanding, addressing the gap where multi-user perspectives and evolving discourse complicate interpretation. The framework combines data-curation pipelines with two core evaluators, CONSINT-Tree for depth/breadth and CONSINT-RAG for correctness, complemented by informativeness metrics like lexical richness and semantic redundancy. Across 20 LLMs spanning proprietary and open-source families, reasoning models generally outperform general ones, but a persistent gap remains between closed and open models and in deep/broad intent comprehension. The work demonstrates the benchmark's scalability, live-update capability, and potential to drive improvements in how LLMs understand nuanced consumer discussions in real-world settings.

Abstract

Understanding human intent is a complex, high-level task for large language models (LLMs), requiring analytical reasoning, contextual interpretation, dynamic information aggregation, and decision-making under uncertainty. Real-world public discussions, such as consumer product discussions, are rarely linear or involve a single user. Instead, they are characterized by interwoven and often conflicting perspectives, divergent concerns, goals, emotional tendencies, as well as implicit assumptions and background knowledge about usage scenarios. To accurately understand such explicit public intent, an LLM must go beyond parsing individual sentences; it must integrate multi-source signals, reason over inconsistencies, and adapt to evolving discourse, similar to how experts in fields like politics, economics, or finance approach complex, uncertain environments. Despite the importance of this capability, no large-scale benchmark currently exists for evaluating LLMs on real-world human intent understanding, primarily due to the challenges of collecting real-world public discussion data and constructing a robust evaluation pipeline. To bridge this gap, we introduce \bench, the first dynamic, live evaluation benchmark specifically designed for intent understanding, particularly in the consumer domain. \bench is the largest and most diverse benchmark of its kind, supporting real-time updates while preventing data contamination through an automated curation pipeline.

Figures (6)

• Figure 1: Overview of CONSINT-Bench, a large-scale and live benchmark designed to evaluate real-world consumer intent understanding.
• Figure 2: Overview of CONSINT-Bench: It includes over 200k product-level discussions across 9 major domains, 54 sub-domains, and more than 1,400 products.
• Figure 3: The overall pipeline of CONSINT-Bench, covering data curation and evaluation. Data Curation: We provide two methods (keyword and semantic) to search and retrieve product discussions daily. It consists of over 200k product-level discussions across 9 major domains, 54 sub-domains, and more than 1,400 products. The benchmark focuses on four primary dimensions and evaluates intent depth across five hierarchical difficulty levels. Afterthat, rule-based and LLM-based filtering is applied to remove irrelevant discussions and retain relevant ones. Evaluation: We propose CONSINT-Tree for more accurate assessment of depth and breadth dimensions, and CONSINT-RAG for correctness. For informativeness, we assess lexical richness and semantic redundancy.
• Figure 4: Lighted Tree from GPT-5 on Google Nest Smart Speaker.
• Figure 5: Lighted Tree from GPT-o3 on Google Nest Smart Speaker.