HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination

TL;DR

HypoSpace addresses the challenge of underdetermination in scientific reasoning by treating LLMs as samplers over finite admissible hypothesis sets $\mathcal{H}_O$ and evaluating them with deterministic validators along three signals: Validity, Uniqueness, and Recovery. It instantiates this framework in three domains with exactly enumerated hypothesis spaces—causal graphs from perturbations, gravity-constrained 3D voxel reconstructions, and Boolean genetic interactions—to quantify how well models cover admissible explanations rather than just produce correct answers. Across instruction-tuned and reasoning-oriented LLMs, the study finds that while Validity remains high, Uniqueness and Recovery degrade as $|\mathcal{H}_O|$ grows, evidencing mode collapse and incomplete exploration despite correct outputs. The work contributes a diagnostic, non-leaderboard framework with deterministic validation and information-theoretic analyses that illuminate exploration dynamics and guide future methods for entropy-aware decoding and memory-informed sampling to better map admissible explanation spaces in scientific tasks.

Abstract

As language models are increasingly used in scientific workflows, evaluating their ability to propose sets of explanations-not just a single correct answer-becomes critical. Many scientific problems are underdetermined: multiple, mechanistically distinct hypotheses are consistent with the same observations. We introduce HypoSpace, a diagnostic suite that treats LLMs as samplers of finite hypothesis sets and measures three complementary indicators: Validity (precision of proposals consistent with observations), Uniqueness (non-redundancy among proposals), and Recovery (coverage of the enumerated admissible set). We instantiate HypoSpace in three structured domains with deterministic validators and exactly enumerated hypothesis spaces: (i) causal graphs from perturbations, (ii) gravity-constrained 3D voxel reconstruction from top-down projections, and (iii) Boolean genetic interactions. Across instruction-tuned and reasoning-focused models, Validity often remains high while Uniqueness and Recovery degrade as the admissible space grows, revealing mode collapse that is invisible to correctness-only metrics. HypoSpace offers a controlled probe-rather than a leaderboard-for methods that explicitly explore and cover admissible explanation spaces. Code is available at: https://github.com/CTT-Pavilion/_HypoSpace.

Figures (4)

• Figure 1: HypoSpace evaluation framework and model performance comparison. (a) Our diagnostic approach treats LLMs as samplers over hypothesis spaces. Given observations $O$, models generate $N$ hypotheses that are validated for consistency with data and deduplicated for uniqueness. We measure three complementary indicators: Validity (VR: precision of valid hypotheses), Uniqueness (NR: non-redundancy among proposals), and Recovery (RR: coverage of the enumerated admissible set $\mathcal{H}_O$). These operationalize creativity theory's appropriateness, originality, and fluency dimensions, respectively. (b) Recovery Rate comparison across models on task of Boolean genetic interactions, showing systematic degradation as hypothesis spaces grow from simple to hard settings, with reasoning models generally outperforming non-reasoning models.
• Figure 2: HypoSpace task instantiations. (a) Causal inference from perturbations: Given intervention observations (e.g., perturbing node C affects nodes B), models propose causal DAGs consistent with the data. Multiple valid graph structures can explain the same perturbation patterns. (b) 3D voxel reconstruction under gravity: From top-down 2D projections, models reconstruct 3D voxel structures satisfying projection constraints and gravity (stacking rules). The same projection admits multiple valid 3D configurations. (c) Boolean genetic interactions: From phenotype observations of parental combinations, models propose Boolean expressions relating inputs to outputs. Expressions that collapse to the same form under our restricted canonicalizer count as one hypothesis. Each domain exemplifies scientific underdetermination where multiple mechanistically distinct hypotheses explain identical observations, enabling measurement of Validity, Uniqueness, and Recovery across enumerated hypothesis spaces.
• Figure 3: Information-theoretic analysis of hypothesis space exploration in HypoSpace.(a) Cumulative entropy as a function of sequential queries, measuring how uncertainty reduction varies across models as they sample from the hypothesis space. Steeper curves indicate more efficient exploration of distinct valid hypotheses. (b) Information gain per query (with 0.5 vertical offset for visual clarity). The curve shape reflects each model's marginal contribution of new information with additional samples, revealing differences in exploration strategies and susceptibility to mode collapse. Models with flatter curves show diminishing returns in hypothesis diversity, consistent with the Recovery Rate degradation observed in Table \ref{['Tab_task']}.
• Figure 4: Failure modes and exploration patterns in HypoSpace hypothesis generation.(a) Error categorization across models: Distribution of failure types including parse failures (malformed outputs), constraint violations (structurally invalid hypotheses), invalid generations (inconsistent with observations), and duplicates (exact and canonical equivalences). Higher proportions of duplicates indicate mode collapse tendencies. (b) Hypothesis space coverage: Fraction of the enumerated admissible set $\mathcal{H}_O$ explored versus unexplored by each model. Limited exploration (lower blue bars) corresponds to reduced Recovery Rates and reveals the extent to which models fail to map the full space of valid explanations, even when maintaining high Validity.