Scientific judgment drifts over time in AI ideation

TL;DR

The paper investigates whether scientists' evaluations of early-stage ideas drift over time, challenging the assumption that human judgments are stationary gold standards for AI ideation. Using a two-wave, within-subject design with 57 participants and 7,182 ratings, the authors show a measurable, modest upward drift in Overall Quality for the same control idea (mean change $+0.61$ on a $0$–$10$ scale, $P = 0.005$), while the relative weighting of evaluation dimensions remains stable. They align an LLM-based evaluator to Wave 1 ratings and reapply it to select Wave 2 ideas, finding that the apparent gains from fixed-snapshot alignment disappear after accounting for drift via a control anchor. The study emphasizes drift-aware evaluation protocols, including repeated controls and longitudinal benchmarks, as essential for building AI ideation systems that truly augment evolving expert judgment, not merely overfit to a transient standard.

Abstract

Scientific discovery begins with ideas, yet evaluating early-stage research concepts is a subtle and subjective human judgment. As large language models (LLMs) are increasingly tasked with generating scientific hypotheses, most systems assume that scientists' evaluations form a fixed gold standard, and that scientists' judgments do not change. Here we challenge this assumption. In a two-wave study with 7,182 ratings from 57 active researchers across six scientific departments, each participant repeatedly evaluated a constant "control" research idea alongside AI-generated ideas. We show that scientists' ratings of the very same idea systematically drift over time: overall quality scores increased by 0.61 points on a 0-10 scale (P = 0.005), and test-retest reliability was only moderate across core dimensions of scientific value, revealing systematic temporal drift in perceived idea quality. Yet the internal structure of judgment remained stable, such as the relative importance placed on originality, feasibility, clarity. We then aligned an LLM-based ideation system to first-wave human ratings and used it to select new ideas. Although alignment improved agreement with Wave-1 evaluations, its apparent gains disappeared once drift in human standards was accounted for. Thus, tuning to a fixed human snapshot produced improvements that were transient rather than persistent. These findings reveal that human evaluation of scientific ideas is not static but a dynamic process with stable priorities and requires shifting calibration. Treating one-time human ratings as immutable ground truth risks overstating progress in AI-assisted ideation and obscuring the challenge of co-evolving with changing expert standards. Drift-aware evaluation protocols and longitudinal benchmarks may therefore be essential for building AI systems that reliably augment, rather than overfit to, human scientific judgment.

Figures (6)

• Figure 1: Experimental design: two-wave evaluation with repeated controls and drift-aware analysis. (A) Each scientist completed two evaluation waves. In each wave, they rated six AI-generated research ideas and one repeated human-written idea (control). The repeated control provides a direct measure of test–retest reliability and temporal drift in idea evaluation. (B) For every idea, participants rated eight scientific-value dimensions (e.g., originality, clarity, effectiveness) and an Overall Quality score (0–10 scale). We inferred the implicit weight assigned to each dimension by regressing Overall Quality on the dimension scores. (C) After Wave 1, human ratings were used to align an LLM-based evaluator to participants' preferences. This aligned evaluator then selected the six highest-ranked AI ideas for each scientist in Wave 2. (D) To assess whether alignment produced true improvement rather than overfitting to a transient snapshot of human criteria, we compared Wave 2 and Wave 1 ratings of AI ideas both naively and using a drift-corrected difference-in-differences anchored on the repeated control idea.
• Figure 2: Scientists’ evaluations of the same scientific idea drift over time. (A) Test-retest reliability of Overall Quality score for the repeated human-written control idea was moderate, indicating non-trivial variability even for identical stimuli. (B) Bland-Altman analysis of Overall Quality shows a systematic upward shift in ratings between waves with no evidence of proportional bias. (C) Test-retest reliability of 8 scientific value dimensions (quadratic weighted kappa with 95% CI). (D) Distribution of within-participant changes (Wave 2 – Wave 1) for the control idea shows a significant group-level increase (mean = +0.61, P = 0.005), while most individual changes fall within a wide variability range, consistent with subtle but systematic drift. (E) Paired trajectories for each participant across both waves illustrate consistent upward movement in Overall Quality and small positive shifts across most dimensions.
• Figure 3: Determinants and structure of scientific judgment across time. (A) Mixed-effects regression examining factors associated with changes in ratings of the unchanged control idea. The only significant predictor was Wave, indicating a systematic upward drift in perceived idea quality not explained by order, contextual comparison, time gap, time of day, or expertise. Self-reported evaluation weights (S-wave) strongly predicted ratings, confirming internal consistency in participants’ stated rubrics. (B) Mixed-effects model of Overall Quality predicted by 8 dimensions, revealing the hierarchy of criteria driving scientific evaluation. (C) Wave $\times$ Dimension interaction terms reveal no significant changes in dimension weights over time, indicating that while absolute ratings drifted, the underlying structure of scientific judgment remained stable.
• Figure 4: Alignment to a snapshot of human judgment yields transient rather than persistent gains. (A) Aligning the LLM evaluator to Wave 1 human ratings substantially reduced prediction error on both training and held-out validation ideas, compared with a uniform-weights baseline. (B) Alignment increased agreement with human evaluations in Wave 1, but this improvement weakened in Wave 2, indicating reduced alignment once human standards drifted. (C) Raw human ratings initially suggested improvement: Wave 2 AI ideas scored 0.28 points higher than Wave 1 ideas, but note that the repeated human control idea increased by 0.61 points over the same period, revealing concurrent upward drift. (D) Difference-in-differences after drift correction: no persistent gain beyond the control’s drift.
• Figure S1: Department Compatibility Matrix