Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement

TL;DR

RefineLab introduces a budget-aware, LLM-driven framework to refine QA datasets by optimally selecting editing operations that improve topic coverage, difficulty balance, and factual integrity under a token budget. It formulates refinement as an ILP that maximizes estimated quality gains while respecting resource constraints, supported by modular operators (Coverage Refiners, Difficulty Calibrators) and a validation pipeline combining PoT and RAG. Empirical results across multiple benchmarks show substantial reductions in distributional gaps, improved distractor quality, high correction rates, and measurable gains in LLM evaluation and fine-tuning performance. The work establishes a scalable path toward reproducible, high-quality dataset design with broad implications for LLM evaluation and domain-specific AI applications.

Abstract

High-quality Question-Answer (QA) datasets are foundational for reliable Large Language Model (LLM) evaluation, yet even expert-crafted datasets exhibit persistent gaps in domain coverage, misaligned difficulty distributions, and factual inconsistencies. The recent surge in generative model-powered datasets has compounded these quality challenges. In this work, we introduce RefineLab, the first LLM-driven framework that automatically refines raw QA textual data into high-quality datasets under a controllable token-budget constraint. RefineLab takes a set of target quality attributes (such as coverage and difficulty balance) as refinement objectives, and performs selective edits within a predefined token budget to ensure practicality and efficiency. In essence, RefineLab addresses a constrained optimization problem: improving the quality of QA samples as much as possible while respecting resource limitations. With a set of available refinement operations (e.g., rephrasing, distractor replacement), RefineLab takes as input the original dataset, a specified set of target quality dimensions, and a token budget, and determines which refinement operations should be applied to each QA sample. This process is guided by an assignment module that selects optimal refinement strategies to maximize overall dataset quality while adhering to the budget constraint. Experiments demonstrate that RefineLab consistently narrows divergence from expert datasets across coverage, difficulty alignment, factual fidelity, and distractor quality. RefineLab pioneers a scalable, customizable path to reproducible dataset design, with broad implications for LLM evaluation.

Figures (14)

• Figure 1: An example of topic coverage (left) and difficulty level distribution (right) in the College Chemistry subset of MMLU before (Original) and after refinement (Refined). Targets follow MIT’s undergraduate chemistry major curriculum and feature a stress-testing difficulty profile.
• Figure 2: Top: the framework of RefineLab. Bottom: the Refinement Operations.
• Figure 3: Comparison of LLM performance on original and refined datasets. Left: GSM8K (original vs. refined). Right: HumanEval (original vs. refined).
• Figure 4: Performance breakdown on original and refined GSM8K (left), and refined dataset composition (right).
• Figure 5: Performance variation on GSM8K across prompting strategies, comparing original vs. refined datasets.