SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence

TL;DR

SciIF introduces a constraint-grounded, auditable benchmark for scientific instruction following across multiple disciplines, addressing the gap where models can be correct yet non-compliant with scientific norms. By defining three pillars—scientific conditions, semantic stability, and specific processes—and a fixed catalog of ten atomic constraints, SciIF enables fine-grained diagnostic evaluation via a generate--then--audit protocol and dual-judge scoring on correctness and constraint compliance. The paper demonstrates that post-training with SciIF data (SFT) and verifier-based RL improves both general instruction following (IFEval) and domain-specific reasoning (MMLU-Physics), and shows transfer to external benchmarks, indicating constraint discipline generalizes beyond SciIF. A key finding is the persistent gap between correctness and constraint adherence, amplified under multi-constraint scenarios, revealing a compositional bottleneck in maintaining multiple scientific requirements. These results advance the quest for reliable, scientifically rigorous AI agents and provide a pathway to extend the framework to multimodal and iterative scientific tasks.

Abstract

As large language models (LLMs) transition from general knowledge retrieval to complex scientific discovery, their evaluation standards must also incorporate the rigorous norms of scientific inquiry. Existing benchmarks exhibit a critical blind spot: general instruction-following metrics focus on superficial formatting, while domain-specific scientific benchmarks assess only final-answer correctness, often rewarding models that arrive at the right result with the wrong reasons. To address this gap, we introduce scientific instruction following: the capability to solve problems while strictly adhering to the constraints that establish scientific validity. Specifically, we introduce SciIF, a multi-discipline benchmark that evaluates this capability by pairing university-level problems with a fixed catalog of constraints across three pillars: scientific conditions (e.g., boundary checks and assumptions), semantic stability (e.g., unit and symbol conventions), and specific processes(e.g., required numerical methods). Uniquely, SciIF emphasizes auditability, requiring models to provide explicit evidence of constraint satisfaction rather than implicit compliance. By measuring both solution correctness and multi-constraint adherence, SciIF enables finegrained diagnosis of compositional reasoning failures, ensuring that LLMs can function as reliable agents within the strict logical frameworks of science.

Figures (8)

• Figure 1: Overview of the SciIF Benchmark. a) Existing answer-only scientific benchmarks evaluate only final answer matching, overlooking scientific constraints and instruction compliance. b) SciIF advances this paradigm by introducing explicit scientific constraints and structured verification rubrics, enabling fine-grained assessment of scientific constraint-aware reasoning. c) SciIF provides constraint-grounded SFT and RL datasets, showing that scientific instruction following capability systematically improves correctness, rigor, and instruction adherence.
• Figure 2: Overview of SciIF. We curate scientific QA data from multiple sources and apply a four-stage human-in-the-loop process to produce well-posed problems paired with explicit scientific constraints and auditable evidence checklists. For evaluation, prompt builders generate answer-generation and per-constraint judge prompts, and two independent model judges audit both answer correctness and constraint compliance under Strict or Loose policies. The example highlights a core failure mode: an answer can match the reference numerically yet fail compliance when required evidence such as units and a one-line unit check is missing.
• Figure 3: Constraint composition of SciIF. Top-right: distribution of the number of enabled constraints per problem. Left: hierarchical breakdown of the constraint catalog, with inner wedges for the three constraint families and outer wedges for the 10 atomic constraints; wedge areas are proportional to how often each constraint is enabled in the test set.
• Figure 4: Comparison of three metrics: answer correctness, single-constraint pass, and multi-constraint overall pass. Single-constraint pass is computed on the 60 items with exactly one enabled constraint. Multi-constraint overall pass is computed on the 274 items with multiple enabled constraints, where an item passes only if all enabled constraints pass.
• Figure 5: Compositional collapse under increasing constraint load: strict compliance rate versus the number of enabled constraints $k$ for ten representative models.