What Has Been Lost with Synthetic Evaluation?

TL;DR

The paper investigates whether LLMs can reliably generate evaluation benchmarks that are as challenging as human-authored ones. By studying CondaQA (negation reasoning) and DROP (discrete/numeric reasoning), it develops a prompting workflow to create Q&A content and contrastive edits, then compares synthetic data to human-authored baselines. The findings show that while generated benchmarks can be largely valid and cheaper to produce, they are generally easier for models and can fail to preserve the relative difficulty or rankings of models. The authors argue for cautious use of synthetic evaluation data, emphasizing the need for human-in-the-loop validation and targeted prompting strategies to maintain benchmark quality for assessing real-world generalization and nuanced reasoning.

Abstract

Large language models (LLMs) are increasingly used for data generation. However, creating evaluation benchmarks raises the bar for this emerging paradigm. Benchmarks must target specific phenomena, penalize exploiting shortcuts, and be challenging. Through two case studies, we investigate whether LLMs can meet these demands by generating reasoning over-text benchmarks and comparing them to those created through careful crowdsourcing. Specifically, we evaluate both the validity and difficulty of LLM-generated versions of two high-quality reading comprehension datasets: CondaQA, which evaluates reasoning about negation, and DROP, which targets reasoning about quantities. We find that prompting LLMs can produce variants of these datasets that are often valid according to the annotation guidelines, at a fraction of the cost of the original crowdsourcing effort. However, we show that they are less challenging for LLMs than their human-authored counterparts. This finding sheds light on what may have been lost by generating evaluation data with LLMs, and calls for critically reassessing the immediate use of this increasingly prevalent approach to benchmark creation.

Figures (26)

• Figure 1: The percentage of valid questions in a sample of 16 for intermediate evaluations, and 100 or 114 for final evaluations. Questions are generated with gpt-4-turbo-2024-04-09 and assessed by one author of this paper. The full prompt code and templates for each iteration can be found in our https://github.com/utahnlp/eval-synth-eval repository.
• Figure 2: The percentage of valid edits in a sample of 16 for intermediate evaluations, and 100 or 114 for final evaluations. Edits are generated with gpt-4-turbo-2024-04-09 and assessed by one author of this paper. The full prompt code and templates for each iteration can be found in our https://github.com/utahnlp/eval-synth-eval repository.
• Figure 3: Illustration of our synthetic dataset creation process. Purple borders around a block indicates relevance to CondaQA, teal borders indicate relevance to DROP. All other blocks are relevant for both datasets.
• Figure 4: Distribution of lengths of model-generated questions vs human-authored questions. We find that on average generated questions are 18.35 words, and the human-authored questions were 27.17 words long. We also find that 26.3% of generated questions are more than 20 words long, but 67.68% of human-authored questions were more than 20 words long.
• Figure 5: The instructions for the CondaQA preference study (§\ref{['sec:pref_study']}).