Data-Model Co-Evolution: Growing Test Sets to Refine LLM Behavior

TL;DR

The paper addresses the friction between data curation and model refinement in rapid LLM development by proposing data-model co-evolution, where a living test set and prompt instructions evolve together. It introduces an interactive system that guides edge-case discovery, rationales, neighborhood probing, and regression testing against a growing test set, with LLM-assisted, human-governed decision making. A formative study informs the design, and a controlled lab study (with twelve participants) shows the workflow yields longer, more explicit prompts, larger test sets, more consistent changes to model behavior, and higher user satisfaction than a Baseline prompt-editing tool. This approach promises more robust, policy-aligned, and accessible LLM applications by embedding continuous, human-in-the-loop test-set growth into the development process, enabling diverse developers to codify local norms and reduce brittle behavior.

Abstract

A long-standing challenge in machine learning has been the rigid separation between data work and model refinement, enforced by slow fine-tuning cycles. The rise of Large Language Models (LLMs) overcomes this historical barrier, allowing applications developers to instantly govern model behavior by editing prompt instructions. This shift enables a new paradigm: data-model co-evolution, where a living test set and a model's instructions evolve in tandem. We operationalize this paradigm in an interactive system designed to address the critical challenge of encoding subtle, domain-specific policies into prompt instructions. The system's structured workflow guides people to discover edge cases, articulate rationales for desired behavior, and iteratively evaluate instruction revisions against a growing test set. A user study shows our workflow helps participants refine instructions systematically and specify ambiguous policies more concretely. This work points toward more robust and responsible LLM applications through human-in-the-loop development aligned with local preferences and policies.

Figures (9)

• Figure 1: System interface for the Co-Evolution Workflow. The left panel supports authoring and tracking prompt instructions across versions. The right panel manages data: the top section maintains a living test set with model responses from both prior and current prompt instruction versions, while the bottom section displays generated candidate inputs for user review. Together, these panels embody the co-evolution principle by tightly coupling specification revisions with test set examples. The examples in this figure, originally written in Korean, were used in the user study and translated into English.
• Figure 2: Overview of the Co-Evolution workflow: (1) the system synthesizes potential user inputs that are likely hard or ambiguous; (2) users examine these cases, label failures, and provide rationales as hypotheses for model errors; (3) neighborhood probing generates similar inputs to test whether the rationale generalizes; (4) the system proposes a revised prompt instruction grounded in the rationale and neighborhood labels; and (5) the expanded test set evaluates model responses across both prior and updated prompt instruction versions. This structured loop scaffolds the shift from a single error to a broader policy rule.
• Figure 3: Neighborhood validation step in the workflow. The top section shows the original selected example, along with the user’s label and written rationale. Below, the system provides semantically similar examples that vary slightly from the original, enabling the user to test whether their rationale holds across a neighborhood of cases. As users label these examples, they actively reflect on the policy boundary, and the results directly inform the next step of revising the prompt instruction. The validated examples are then added to the test set, where they continue to be used in subsequent evaluations.
• Figure 4: Length of prompt instructions created under each condition, measured in word count and sentence count. Each point represents one participant (jittered for visibility), with box plots summarizing distributions. Instructions produced with the Co-Evolution workflow were significantly longer than those from the Baseline.
• Figure 5: Comparison of how prompt instructions across conditions provided specific information about policy. From left to right: (1) number of policy statements included, (2) whether explicit exceptions or constraints were stated, and (3) whether concrete examples were given. While the number of policy statements was similar, Co-Evolution led participants to add more examples to illustrate.