Are Large Language Models Good Statisticians?

TL;DR

The paper introduces StatQA, a benchmark designed to evaluate whether Large Language Models can perform statistical analysis tasks by selecting relevant data columns and assessing the applicability of statistical methods for hypothesis testing. It uses a reverse-synthesis pipeline to generate 11,623 examples from real-world tabular data, with a mini-StatQA subset for cost-efficient testing, and a quality-control process including template design and expert reviews. Across open-source and proprietary LLMs, results show substantial gaps, with GPT-4o achieving up to approximately 64.83% accuracy; domain knowledge prompts and fine-tuning improve performance but do not reach human-level proficiency, especially on complex tasks like contingency table and variance tests. Comparative human experiments reveal distinct error patterns from LLMs, suggesting potential for beneficial human-AI collaboration to strengthen statistical reasoning in practice. The work highlights opportunities to improve LLM reasoning about methodological applicability, expand the benchmark, and explore collaborative strategies to enhance statistics-focused AI tools.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across a range of scientific tasks including mathematics, physics, and chemistry. Despite their successes, the effectiveness of LLMs in handling complex statistical tasks remains systematically under-explored. To bridge this gap, we introduce StatQA, a new benchmark designed for statistical analysis tasks. StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods. We systematically experiment with representative LLMs using various prompting strategies and show that even state-of-the-art models such as GPT-4o achieve a best performance of only 64.83%, indicating significant room for improvement. Notably, while open-source LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit marked improvements, outperforming all in-context learning-based methods (e.g. GPT-4o). Moreover, our comparative human experiments highlight a striking contrast in error types between LLMs and humans: LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors. This divergence highlights distinct areas of proficiency and deficiency, suggesting that combining LLM and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. Our source code and data are available at https://statqa.github.io/.

Figures (11)

• Figure 1: An Example of Statistical Analysis Task
• Figure 2: An Example in StatQA. (1) Dataset ($D$) is the datasets for statistical tasks; (2) Question ($Q^*(\mathbb{C})$) is a statistical question, refined by GPT-3.5-Turbo; (3) Task indicates the statistical task category; (4) Difficulty includes easy and hard levels. (5) Relevant Columns ($\mathbb{C}$) are data columns relevant to the statistical question; (6) Results ($\mathbb{R}$) include applicable statistical methods ($\mathbb{M}$), the computational results, and preliminary conclusions; (7) Ground Truth includes relevant columns ($\mathbb{C}$) and all applicable methods ($\mathbb{M}$) for this statistical task;
• Figure 3: The Proportion of Statistical Tasks in StatQA. The inner ring represents the five types of statistical tasks mentioned, while the outer ring corresponds to the methods associated with each task.
• Figure 4: The Pipeline for Synthesizing StatQA
• Figure 5: Best Results of Each Model in Five Sub-tasks (Section \ref{['task_scope']}).