Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making

Nazia Riasat

Abstract

Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific workflows, where correctness and validity are critical. However, evaluation practices often emphasize stability or reproducibility across repeated runs. While these properties are desirable, stability alone does not guar- antee agreement with statistical ground truth when such references are available. We introduce a controlled behavioral evaluation framework that explicitly sep- arates four dimensions of LLM decision-making: stability, correctness, prompt sensitivity, and output validity under fixed statistical inputs. We evaluate multi- ple LLMs using a statistical gene prioritization task derived from differential ex- pression analysis across prompt regimes involving strict and relaxed significance thresholds, borderline ranking scenarios, and minor wording variations. Our ex- periments show that LLMs can exhibit near-perfect run-to-run stability while sys- tematically diverging from statistical ground truth, over-selecting under relaxed thresholds, responding sharply to minor prompt wording changes, or producing syntactically plausible gene identifiers absent from the input table. Although sta- bility reflects robustness across repeated runs, it does not guarantee agreement with statistical ground truth in structured scientific decision tasks. These findings highlight the importance of explicit ground-truth validation and output validity checks when deploying LLMs in automated or semi-automated scientific work- flows.

When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making

Abstract

Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific workflows, where correctness and validity are critical. However, evaluation practices often emphasize stability or reproducibility across repeated runs. While these properties are desirable, stability alone does not guar- antee agreement with statistical ground truth when such references are available. We introduce a controlled behavioral evaluation framework that explicitly sep- arates four dimensions of LLM decision-making: stability, correctness, prompt sensitivity, and output validity under fixed statistical inputs. We evaluate multi- ple LLMs using a statistical gene prioritization task derived from differential ex- pression analysis across prompt regimes involving strict and relaxed significance thresholds, borderline ranking scenarios, and minor wording variations. Our ex- periments show that LLMs can exhibit near-perfect run-to-run stability while sys- tematically diverging from statistical ground truth, over-selecting under relaxed thresholds, responding sharply to minor prompt wording changes, or producing syntactically plausible gene identifiers absent from the input table. Although sta- bility reflects robustness across repeated runs, it does not guarantee agreement with statistical ground truth in structured scientific decision tasks. These findings highlight the importance of explicit ground-truth validation and output validity checks when deploying LLMs in automated or semi-automated scientific work- flows.
Paper Structure (41 sections, 2 equations, 5 figures, 2 tables)

This paper contains 41 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup and Evaluation
  3. Results
  4. Stability Does Not Imply Correctness across LLMs
  5. Prompt Regime and Wording Sensitivity under identical inputs
  6. Over-selection under relaxed thresholds
  7. Hallucinated gene identifiers
  8. Discussion and Conclusion
  9. Prompt Specifications
  10. Common Input Format
  11. Group 1: Strict Significance Filtering
  12. Prompt P1 (FDR $\leq$ 0.05).
  13. Group 2: Relaxed / Borderline Significance
  14. Prompt P5 (0.05 $<$ FDR $\leq$ 0.10).
  15. Group 3: Ranked Prioritization of Borderline Genes
  16. ...and 26 more sections

Figures (5)

  • Figure 1: Failure modes in LLM-based statistical gene prioritization under fixed input data. Despite high internal stability, models may disagree with the DESeq2-derived statistical reference (B), exhibit prompt sensitivity (C), or produce invalid gene identifiers (D), Panel (E) summarizes observed behaviors.
  • Figure 2: Stability does not imply correctness. Within-LLM stability (pairwise Jaccard similarity across repeated runs; x-axis) versus agreement with the DESeq2-derived statistical reference (Jaccard against ground truth; y-axis). Each point represents the mean stability and agreement values aggregated across 10 repeated runs per configuration. Run-level variability was minimal due to near-perfect within-model stability.
  • Figure 3: LLM behavior across prompt regimes. Mean Jaccard similarity to DESeq2 reference of Top-20 gene sets across threshold-based selection, borderline prioritization, and prompt wording variants.
  • Figure 4: Prompt sensitivity under minor wording changes (Prompt 7a vs. 7b). Jaccard similarity and overlap coefficient between Top-20 gene sets produced under semantically similar prompts differing only in emphasis. Both metrics exhibit the same qualitative pattern, indicating that prompt wording can alter selected gene sets even when accounting for differences in set containment.
  • Figure 5: Evaluates validity in ranked gene lists. ChatGPT and Gemini consistently return identifiers present in the input table. In contrast, Claude frequently outputs ranked lists composed largely of identifiers absent from the input table under this prompt configuration.