InfoSynth: Information-Guided Benchmark Synthesis for LLMs

TL;DR

InfoSynth tackles benchmark creation for LLM evaluation by introducing an information-theoretic framework that quantifies novelty with $D_{KL}(q\|p)$ and diversity with the differential entropy $h(p)$ of problem embeddings in $\mathbb{R}^d$. It provides an end-to-end pipeline that uses genetic algorithms, mutation/crossover, iterative code feedback, deduplication, and postprocessing to synthesize robust Python coding problems with executable test-and-solution verification. Empirical results show generated benchmarks achieve higher novelty and diversity than seed data and maintain high correctness (≈$97\%$) while enabling explicit control over novelty/diversity and difficulty. The work offers a scalable, self-verifying framework for contamination-free LLM evaluation benchmarks and demonstrates practical benefits for robust reasoning and code-generation assessment.

Abstract

Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation. However, efficiently creating new benchmarks to evaluate these capabilities remains a challenge. Traditional benchmark creation relies on manual human effort, a process that is both expensive and time-consuming. Furthermore, existing benchmarks often contaminate LLM training data, necessitating novel and diverse benchmarks to accurately assess their genuine capabilities. This work introduces InfoSynth, a novel framework for automatically generating and evaluating reasoning benchmarks guided by information-theoretic principles. We propose metrics based on KL-divergence and entropy to quantify benchmark novelty and diversity without relying on costly model evaluations. Building on this framework, we develop an end-to-end pipeline that synthesizes robust Python coding problems from seed datasets using genetic algorithms and iterative code feedback. Our method generates accurate test cases and solutions to new problems 97% of the time, and the synthesized benchmarks consistently exhibit higher novelty and diversity compared to their seed datasets. Moreover, our algorithm provides a method for controlling the novelty/diversity and difficulty of generated problems. InfoSynth offers a scalable, self-verifying pipeline for constructing high-quality, novel and diverse benchmarks for LLMs. Project Page: https://ishirgarg.github.io/infosynth_web/

Figures (10)

• Figure 1: Left: The full Leetcode dataset has higher novelty than its Hash Table and String subsets as expected. Middle: The MBPP dataset has high novelty against the Leetcode dataset, whereas the Leetcode subdatasets have very little relative novelty as expected. Right: Leetcode and APPS have high novelty as their problems are harder and very different from MBPP, whereas HumanEval has low novelty as it is known to be more similar to MBPP. All plots show 95% confidence intervals.
• Figure 2: Left: Leetcode vs. MBPP entropy. MBPP shows lower entropy due to simpler, more repetitive problems. Middle: Codeforces vs. subsets. Full datasets have higher entropy than topic-specific subsets, except Math, which overlaps with others (e.g., DP, Greedy) and thus appears highly dispersed when isolated. Right: Leetcode and APPS have high diversity as their problems span many different computer science topics, whereas HumanEval has low diversity as the problems are easier and known to be more similar to MBPP. All plots show 95% confidence intervals.
• Figure 3: Generation Pipeline. Each colony receives a subset of the seed problems and applies mutation or crossover to them at each iteration. For each problem, it generates solutions and tests which go through multiple iterations of testing to ensure correctness. Deduplication removes similar problems within each colony; the remaining ones are used as seed data for the next iteration. The colony outputs are merged and deduplicated to produce the final dataset.
• Figure 4: Novelty and diversity analysis across MBPP and Leetcode variants.
• Figure 5: Fraction of problems relating to each topic for the 10 most common topics