FEval-TTC: Fair Evaluation Protocol for Test-Time Compute

TL;DR

FEval-TTC tackles the instability of LLM performance and API pricing by providing a fair, reproducible evaluation protocol for test-time compute. It introduces an open-source, modular framework that uses pre-recorded CoTs, standardized prompts, and a unified dollar-cost model to compare TTC methods across multiple LLM families and reasoning datasets. The approach enables rapid, cost-efficient benchmarking of methods like Self-Consistency, Best-of-N, and cascade strategies by replacing live API calls with pre-recorded responses, yielding reproducible results despite evolving pricing. This work empowers researchers with robust, scalable evaluation, reducing both time and monetary overhead while maintaining fair comparisons across changing model and pricing landscapes.

Abstract

The performance of Large Language Models (LLMs) and the associated dollar costs of API calls can fluctuate over time, potentially invalidating conclusions drawn in prior research. To address this, we propose a Fair Evaluation protocol for Test-Time Compute (FEval-TTC), designed to ensure consistent assessment of test-time compute (TTC) methods, regardless of such fluctuations. FEval-TTC focuses on the evaluation of TTC methods that utilize underlying Chains-of-Thought (CoT). It supports evaluations across multiple LLMs on a diverse set of mathematical and commonsense reasoning datasets. The few-shot prompting and answer extraction processes are standardized across datasets, reducing both time and monetary overhead for researchers. Furthermore, we provide a cost modelling procedure that estimates both the token and dollar cost per query, facilitating equitable comparisons of prevalent TTC methods. We open-source FEval-TTC for public use at https://github.com/networkslab/feval_ttc .

Figures (3)

• Figure 1: FEval-TTC comprises of three distinct groups of datasets, each consisting of question–answer pairs (see Section \ref{['sec:datasets']}). Each dataset is queried by multiple LLMs from different families with a standardized query format. We provide 40 sampled Chains-of-Thoughts (CoTs) with extracted answers, number of tokens, and the corresponding dollar cost of inference per question (see Section \ref{['sec:llms']}).
• Figure 2: Evaluation of CoT+SC and Best-of-N algorithms on AQuA dataset using Qwen 32B for varying number of CoTs.
• Figure 3: Evaluation of MoT and ModelSwitch algorithms on MATH-500 dataset using Llama models for varying computational costs.