CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

TL;DR

CricBench introduces a specialized, expert-curated Text-to-SQL benchmark for cricket analytics with English and Hindi queries, built on a normalized IPL database. A Complexity Router and dynamic prompting inject domain rules to assess context-aware reasoning beyond schema linking. DeepSeek R1 sets a new open-weight baseline (≈50.6% data accuracy) and Hindi code-mixing shows competitive or superior performance in some settings, yet large domain gaps remain for general-purpose models like GPT-4o and Claude 3.7. The study highlights the limits of current prompting techniques and the need for deeper domain-grounded reasoning to bridge the gap between general benchmarks and specialized sports analytics.

Abstract

Cricket is the second most popular sport globally, commanding a massive following of over 2.5 billion fans globally. Enthusiasts and analysts frequently seek advanced statistical insights, such as long-term historical performance trends or complex player comparisons, that are often unavailable through standard web searches. While Large Language Models (LLMs) have advanced significantly in Text-to-SQL tasks, their capability to handle the domain-specific nuances, complex schema variations, and multilingual requirements inherent to sports analytics remains under-explored. To investigate this potential capability gap, we present CricBench, a comprehensive benchmark suite for evaluating LLMs on specialized cricket data. To curate a "Gold Standard" dataset, we collaborate with domain experts in cricket and SQL to manually author complex queries, ensuring logical correctness. Recognizing linguistic diversity, we construct the benchmark in both English and Hindi, establishing a framework that is open for further extension to other regional languages. We evaluate six state-of-the-art models, including GPT-4o, Claude 3.7 Sonnet, and open-source models, using a strict evaluation protocol. Our results reveal that high performance on general benchmarks does not guarantee success in specialized domains. While the open-weights reasoning model DeepSeek R1 achieves state-of-the-art performance (50.6%), surpassing proprietary giants like Claude 3.7 Sonnet (47.7%) and GPT-4o (33.7%), it still exhibits a significant accuracy drop when moving from general benchmarks (BIRD) to CricBench. Furthermore, we observe that code-mixed Hindi queries frequently yield parity or higher accuracy compared to English, challenging the assumption that English is the optimal prompt language for specialized SQL tasks.