RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets

TL;DR

RoSE addresses the challenge of selecting the best LLM generator for synthetic-data augmentation when human test sets are unavailable, by a cross-evaluation approach that trains a small model on each generator's data and tests on other generators' synthetic data. It robustly outperforms intrinsic metrics, achieving an average downstream F1 gap of $0.76\%$ from the optimal generator and showing positive correlations with human evaluation, across 11 languages and 3 tasks. The method relies on in-context human examples during data generation and demonstrates resilience across task types, language resources, and varying numbers of candidate LLMs, though it remains computationally intensive. The work contributes a practical, data-efficient tool for low-resource settings and provides code and results publicly to facilitate adoption and further benchmarking.

Abstract

LLMs are powerful generators of synthetic data, which are used for training smaller, specific models. This is especially valuable for low-resource languages, where human-labelled data is scarce but LLMs can still produce high-quality text. However, LLMs differ in how useful their outputs are for training. Selecting the best LLM as a generator is challenging because extrinsic evaluation requires costly human annotations (which are often unavailable for low-resource languages), while intrinsic metrics correlate poorly with downstream performance. We introduce Round robin Synthetic data Evaluation (RoSE), a proxy metric for selecting the best LLM generator without human test sets. RoSE trains a small model on the outputs of a candidate generator (LLM) and then evaluates it on generated synthetic examples from all other candidate LLMs. The final RoSE score is the mean performance of this small model. Across six LLMs, eleven languages, and three tasks (sentiment, topic, intent), RoSE identifies the optimal generator more often than any other intrinsic heuristics. RoSE outperforms intrinsic heuristics and comes within 0.76 percentage points of the optimal generator baseline. This result is measured in terms of downstream performance, obtained by training a small model on the chosen generator's outputs (optimal vs. proxy metric selected) and evaluating it on human-labelled test data. Additionally, RoSE is the only metric to achieve a positive correlation with performance on human test data.

Figures (11)

• Figure 1: Overview of RoSE proxy metric calculations. For each candidate generator (LLM), we first generate synthetic data. Then, we train a smaller model on each generator’s synthetic dataset and evaluate it on the synthetic data generated by all other generators. The mean F1 score across these test sets is the final RoSE score for that LLM. The highest RoSE score LLM is considered the best generator.
• Figure 2: Comparison of proxy metrics for selecting the best LLM generator. Bars show the average gap in mean F1 score for models trained on the best generator selected by metrics vs. the optimal generator (smaller is better). The models are evaluated on human test data. The best metric is green, the second best is orange, and the worst is red.
• Figure 3: Forest plot showing mean Pearson and Rank (Kendal $\tau$) correlations between proxy metrics (except for random, as it has no values) and downstream performance of models on human-labelled data (F1). Error bars denote confidence intervals. RoSE is the most reliable proxy metric; its average Pearson and rank correlations with human F1 are clearly higher than all intrinsic metrics.
• Figure 4: Comparison of a selection of proxy metrics for selecting the best LLM generator when comparing various combinations of LLMs (all combinations of up to 6 LLMs are considered). RoSE is the best proxy metric for a varying number of LLMs compared.
• Figure 5: Number of randomly chosen LLMs used for computing RoSE and its effect on F1 score difference to optimal LLM generator selection. RoSE benefits from more LLMs being used during the evaluation of the downstream classifier during its computation.