EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta

TL;DR

The paper identifies fluent, MC-centric evaluation as insufficient for gauging LLM reasoning and factual accuracy in open-ended tasks. It proposes EQUATOR, a deterministic evaluator that pairs questions with human-evaluated answers stored in a vector database, uses cosine similarity for semantic matching, and leverages smaller LLMs as automated evaluators to enhance scalability. Across two studies, EQUATOR yields significantly lower mean scores than traditional methods (e.g., p < 0.005 or p < 0.0001) with large effect sizes ($d$ values up to 2.85), underscoring how deterministic scoring reduces ambiguity and better captures reasoning failures. The framework also demonstrates token savings and practical implications for high-stakes applications, while outlining biases and mitigation pathways to guide future refinements and broader adoption.

Abstract

Despite the remarkable coherence of Large Language Models (LLMs), existing evaluation methods often suffer from fluency bias and rely heavily on multiple-choice formats, making it difficult to assess factual accuracy and complex reasoning effectively. LLMs thus frequently generate factually inaccurate responses, especially in complex reasoning tasks, highlighting two prominent challenges: (1) the inadequacy of existing methods to evaluate reasoning and factual accuracy effectively, and (2) the reliance on human evaluators for nuanced judgment, as illustrated by Williams and Huckle (2024)[1], who found manual grading indispensable despite automated grading advancements. To address evaluation gaps in open-ended reasoning tasks, we introduce the EQUATOR Evaluator (Evaluation of Question Answering Thoroughness in Open-ended Reasoning). This framework combines deterministic scoring with a focus on factual accuracy and robust reasoning assessment. Using a vector database, EQUATOR pairs open-ended questions with human-evaluated answers, enabling more precise and scalable evaluations. In practice, EQUATOR significantly reduces reliance on human evaluators for scoring and improves scalability compared to Williams and Huckle's (2004)[1] methods. Our results demonstrate that this framework significantly outperforms traditional multiple-choice evaluations while maintaining high accuracy standards. Additionally, we introduce an automated evaluation process leveraging smaller, locally hosted LLMs. We used LLaMA 3.2B, running on the Ollama binaries to streamline our assessments. This work establishes a new paradigm for evaluating LLM performance, emphasizing factual accuracy and reasoning ability, and provides a robust methodological foundation for future research.

Figures (14)

• Figure 1: EQUATOR Evaluator Framework - Drawn by Rahul Murugan
• Figure 2: Cohen’s d Effect Sizes Chart
• Figure 3: Cohen’s d Effect Sizes Chart
• Figure 4: Comparison of Mean Scores, Confidence Interval (CI) Lower Bounds, and CI Upper Bounds for Each Model on the Williams and Huckle 9-13 Multiple-Choice Benchmark. This chart highlights the performance differences when applying the EQUATOR framework, showcasing its stricter scoring criteria than the original evaluation method. The EQUATOR framework results in significantly lower scores compared to the original benchmark for most models, indicating its stricter evaluation standards. Human-evaluated scores are the highest, confirming the limitations of automated evaluation methods.
• Figure 5: The accompanying figure illustrates the Z Interval Error associated with each model as evaluated against the Williams and Huckle 9-13 Multiple-Choice benchmark. This assessment contrasts the results obtained from the EQUATOR framework with those derived from non-EQUATOR evaluations. The Z Interval Error serves as a metric for quantifying the uncertainty inherent in the evaluation scores.