Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets

TL;DR

The paper addresses reproducibility and regression risks in rapidly evolving generative AI by introducing GPR-bench, a regression-testing framework that pairs a bilingual, task-diverse dataset with an automated LLM-as-Judge evaluation pipeline. It formalizes a regression-testing methodology and provides an open-source toolkit for running, recording, and comparing outputs across model versions and prompt configurations. Empirically, it shows that prompt engineering can yield substantial conciseness gains with only modest accuracy costs, while newer model versions provide limited, often non-significant improvements in correctness on the benchmark. The framework emphasizes lightweight extensibility, multilingual coverage, and model-agnostic scoring, offering a practical pathway for continuous, community-driven evaluation and highlighting design considerations for benchmarks in fast-moving language-model landscapes.

Abstract

Reproducibility and reliability remain pressing challenges for generative AI systems whose behavior can drift with each model update or prompt revision. We introduce GPR-bench, a lightweight, extensible benchmark that operationalizes regression testing for general purpose use cases. GPR-bench couples an open, bilingual (English and Japanese) dataset covering eight task categories (e.g., text generation, code generation, and information retrieval) and 10 scenarios in each task categories (80 total test cases for each language) with an automated evaluation pipeline that employs "LLM-as-a-Judge" scoring of correctness and conciseness. Experiments across three recent model versions - gpt-4o-mini, o3-mini, and o4-mini - and two prompt configurations (default versus concise-writing instruction) reveal heterogeneous quality. Our results show that newer models generally improve correctness, but the differences are modest and not statistically significant, suggesting that GPR-bench may not be sufficiently challenging to differentiate between recent model versions. In contrast, the concise-writing instruction significantly enhances conciseness (+12.37 pp, Mann-Whitney U test: p < 0.001, effect size r = 0.2995) with minimal degradations on accuracy (-1.7 pp), demonstrating the effectiveness of prompt engineering. Released under the MIT License, GPR- bench lowers the barrier to initiating reproducibility monitoring and provides a foundation for community-driven extensions, while also raising important considerations about benchmark design for rapidly evolving language models.

Figures (10)

• Figure 1: Conceptual illustration of how system prompt refinements (prompt engineering) could affect answer quality across different task types. (a) Before Improvement: a hypothetical baseline scenario showing mixed performance across tasks. (b) After Improvement: an example of potential differential effects following prompt refinement—demonstrating how improvements in one area (e.g., code generation) might coincide with degradations in others (e.g., persona-based responses). This illustrative case highlights the importance of comprehensive regression testing across diverse task types.
• Figure 2: Comparison of correctness scores for English prompts. Bar graph showing mean correctness scores and standard deviations across different models. Includes overall average (blue line), skill-specific averages (colored dashed lines), and individual prompt data (gray dotted lines). The change in correctness was minimal when the model was changed.
• Figure 3: Comparison of conciseness scores by prompt type for English prompts. Bar graph showing mean conciseness scores and standard deviations for default and concise prompts. Includes overall average (blue line) and model-specific averages (colored dashed lines).
• Figure 4: Statistical comparison of conciseness scores between default and concise prompts. Box plot showing score distributions with individual data points. The Mann-Whitney U test confirmed a statistically significant difference (p < 0.001) with a small effect size (r = 0.2995).
• Figure 5: Comparison of conciseness scores for English prompts. Bar graph showing mean conciseness scores and standard deviations across different models. Includes overall average (blue line), skill-specific averages (colored dashed lines), and individual prompt data (gray dotted lines).