Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets

TL;DR

This work introduces a closed-loop, data-centric approach to constructing Supervised Fine-Tuning datasets for LLMs using OpenDataArena (ODA). It demonstrates how value-anchored, multi-evaluator signals from ODA can guide data sourcing, curation, selection, synthesis, and verification to produce high-quality training data. The authors instantiate two datasets, ODA-Math-460k and ODA-Mixture, with rigorous data pipelines and two budgeting regimes, achieving state-of-the-art performance and superior data efficiency on math and multi-domain benchmarks. The results substantiate a shift toward transparent, evaluation-driven dataset engineering as a core driver of LLM capability and generalization.

Abstract

The construction of Supervised Fine-Tuning (SFT) datasets is a critical yet under-theorized stage in the post-training of Large Language Models (LLMs), as prevalent practices often rely on heuristic aggregation without a systematic understanding of how individual samples contribute to model performance. In this report, we propose a paradigm shift from ad-hoc curation to a closed-loop dataset engineering framework using OpenDataArena (ODA), which leverages value-anchored rankings and multi-dimensional analysis to transform value benchmarking into feedback signals guiding dataset construction. We instantiate this methodology through two new datasets: \textbf{ODA-Math-460k}, a specialized mathematics reasoning dataset that utilizes a novel two-stage difficulty-aware pipeline to achieve State-of-the-Art (SOTA) results on benchmarks such as AIME and HMMT, and \textbf{ODA-Mixture (100k \& 500k)}, a series of multi-domain instruction datasets built via an ``Anchor-and-Patch'' strategy that outperforms significantly larger open-source baselines. Our empirical results demonstrate that ODA-driven datasets significantly improve both domain-specific reasoning and general utility while achieving superior data efficiency, validating a transition toward data-centric AI where transparent evaluation serves as the primary engine for engineering high-quality training data.

Figures (12)

• Figure 1: Performance vs. dataset size. ODA-Math/Mixture show great result with high efficiency.
• Figure 2: Overview of the construction of ODA-Math-460k and ODA-Mixture using OpenDataArena.
• Figure 3: Overview of the construction of ODA-Math-460k.
• Figure 4: Top problem contributors for ODA-Math-460k.
• Figure 5: Comparison on subjects distribution of problems across datasets.