Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains

TL;DR

FARE tackles the need for scalable, reasoning-centric automatic evaluators by scaling data to 2.5M samples across five tasks and multiple domains, training 8B and 20B parameter evaluators with a simple semi-online RS-SFT pipeline. By blending existing high-quality data with synthetic seed data via programmatic error injection and generate-then-grade, the authors enable stable, multi-task learning without teacher models. Empirically, FARE-8B and FARE-20B achieve state-of-the-art open-source evaluation results, outperforming RL-trained judges and large baselines on core benchmarks and delivering tangible gains in downstream use cases such as inference-time reranking, GRPO verification, and domain-specific continual finetuning. This work demonstrates a practical, data-centric blueprint for scalable evaluators in reasoning-focused domains and highlights the value of large-scale, multi-task training for open-source AI evaluation tools.

Abstract

Finetuning specialized generative evaluators has emerged as a popular paradigm to meet the increasing demand for scalable evaluation during both training and test-time. However, recent work has largely focused on applying new methodology, such as reinforcement learning (RL), to training evaluators, shying away from large-scale, data-driven development. In this work, we focus on data scaling, curating a set of 2.5M samples spanning five unique evaluation tasks (pairwise, step-level, reference-free and reference-based verification, and single rating) and multiple domains focused on reasoning evaluation. With our data, we train Foundational Automatic Reasoning Evaluators (FARE), a family of 8B and 20B (with 3.6B active) parameter evaluators, with a simple iterative rejection-sampling supervised finetuning (SFT) approach. FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators, surpassing specialized 70B+ evaluators. Beyond static benchmarks, we evaluate FARE in real-world tasks: As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH. As verifiers in RL training, FARE improves the downstream RL-trained model performance by up to 14.1% vs. string-matching verifiers. When initialized from FARE, a continually-finetuned FARE-Code outperforms gpt-oss-20B by 65% on evaluating test-case quality.

Figures (5)

• Figure 2: Breakdown of our curated training dataset of 2.5M samples by task (left), domain (center), and curation approach (right). Domain breakdown excludes step-level data, which is entirely math.
• Figure 3: Best-of-10 performance for Llama-3.1-8B generator across 4 challenging benchmarks with baseline (red line) and oracle (green line) performance with FARE and SFR-Judge (SFR), Skywork Critic (SC), Compass-Judger (CJ), and Self-Taught Evaluator (STE) as baselines. FARE are the best small ($\leq$14B) and large ($\geq$ 20B) reranking models: FARE-20B achieves near oracle re-ranking performance on MATH, while FARE-8B matches 70B judges.
• Figure 4: Performance of downstream GRPO-trained model with FARE as reference-based verifiers. Moving from string based output matching or weaker verifiers to FARE brings tangible performance gains across both natural-language based (e.g., GPQA) and math settings.
• Figure 5: Continual training of FARE-20B for code evaluation with only 15K samples yields larger gains over gpt-oss-20B/120B.
• Figure 6: Pairwise positional robustness emerges as at large data scales, as shown with training FARE-8B and an earlier development run initialized with Qwen2.5-7B-Instruct.