Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks

Crish Nagarkar, Leonid Bogachev, Serge Sharoff

TL;DR

This study investigates whether LLMs can solve statistical tasks and reliably judge the quality of their reasoning. It trains three open-source 7B LLMs with LoRA-based fine-tuning and 8-bit quantization on a newly curated 2000-question statistical reasoning benchmark, benchmarking against human scores across correctness, step-by-step logic, and explanation quality. It also compares automated evaluation metrics (BLEU, BERTScore, SBERT) with LLM-based judging and finds LLM-as-a-Judge aligns more closely with human judgments, revealing limitations of traditional metrics. The results show architecture-dependent gains, with fine-tuned models approaching statistics-student performance and offering potential for educational technology and data-analysis assistance, while underscoring the need for scalable uncertainty quantification and broader domain coverage.

Abstract

This paper investigates the ability of large language models (LLMs) to solve statistical tasks, as well as their capacity to assess the quality of reasoning. While state-of-the-art LLMs have demonstrated remarkable performance in a range of NLP tasks, their competence in addressing even moderately complex statistical challenges is not well understood. We have fine-tuned selected open-source LLMs on a specially developed dataset to enhance their statistical reasoning capabilities, and compared their performance with the human scores used as a benchmark. Our results show that the fine-tuned models achieve better performance on advanced statistical tasks on the level comparable to a statistics student. Fine-tuning demonstrates architecture-dependent improvements, with some models showing significant performance gains, indicating clear potential for deployment in educational technology and statistical analysis assistance systems. We also show that LLMs themselves can be far better judges of the answers quality (including explanation and reasoning assessment) in comparison to traditional metrics, such as BLEU or BertScore. This self-evaluation capability enables scalable automated assessment for statistical education platforms and quality assurance in automated analysis tools. Potential applications also include validation tools for research methodology in academic and industry settings, and quality control mechanisms for data analysis workflows.

Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks

TL;DR

This study investigates whether LLMs can solve statistical tasks and reliably judge the quality of their reasoning. It trains three open-source 7B LLMs with LoRA-based fine-tuning and 8-bit quantization on a newly curated 2000-question statistical reasoning benchmark, benchmarking against human scores across correctness, step-by-step logic, and explanation quality. It also compares automated evaluation metrics (BLEU, BERTScore, SBERT) with LLM-based judging and finds LLM-as-a-Judge aligns more closely with human judgments, revealing limitations of traditional metrics. The results show architecture-dependent gains, with fine-tuned models approaching statistics-student performance and offering potential for educational technology and data-analysis assistance, while underscoring the need for scalable uncertainty quantification and broader domain coverage.

Abstract

This paper investigates the ability of large language models (LLMs) to solve statistical tasks, as well as their capacity to assess the quality of reasoning. While state-of-the-art LLMs have demonstrated remarkable performance in a range of NLP tasks, their competence in addressing even moderately complex statistical challenges is not well understood. We have fine-tuned selected open-source LLMs on a specially developed dataset to enhance their statistical reasoning capabilities, and compared their performance with the human scores used as a benchmark. Our results show that the fine-tuned models achieve better performance on advanced statistical tasks on the level comparable to a statistics student. Fine-tuning demonstrates architecture-dependent improvements, with some models showing significant performance gains, indicating clear potential for deployment in educational technology and statistical analysis assistance systems. We also show that LLMs themselves can be far better judges of the answers quality (including explanation and reasoning assessment) in comparison to traditional metrics, such as BLEU or BertScore. This self-evaluation capability enables scalable automated assessment for statistical education platforms and quality assurance in automated analysis tools. Potential applications also include validation tools for research methodology in academic and industry settings, and quality control mechanisms for data analysis workflows.
Paper Structure (22 sections, 2 figures, 4 tables)

This paper contains 22 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundations of LLMs
  4. Automatic Evaluation Methods
  5. Application to Statistical Reasoning
  6. Methodology
  7. Data Curation
  8. Dataset Construction
  9. Evaluation and Marking Scheme
  10. Model Training and Fine-Tuning
  11. Parameter-Efficient Fine-Tuning Methodology
  12. Evaluation and Error Analysis
  13. Human Evaluation of Quality
  14. Metric Reliability Assessment
  15. Quality of Automated Metrics
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Example question with reference solution.
  • Figure 2: Comparison of automated metrics, based on their Kendall's $\tau$ correlation with the human scores.