AnesSuite: A Comprehensive Benchmark and Dataset Suite for Anesthesiology Reasoning in LLMs

TL;DR

AnesSuite introduces the first comprehensive benchmark and dataset suite dedicated to anesthesiology reasoning in LLMs, addressing a gap where specialized medical reasoning is underexplored. The framework comprises AnesBench (cross-lingual, multi-level reasoning), AnesCorpus, AnesQA, and AnesR1, together enabling continued pre-training, supervised fine-tuning, and verifiable RL-based training. The Morpheus baseline collection demonstrates that domain-focused data and targeted training can achieve substantial gains—even rivaling larger models—across domain-specific and general benchmarks, while revealing key factors such as model scale, multilingual transfer, and CoT length that shape performance. Through extensive ablations, the work elucidates how training data composition and strategies interact with model capabilities, offering practical guidance for building anesthesiology-reasoning LLMs. AnesSuite and Morpheus will be open-sourced to spur community-driven advances in domain-specific medical reasoning for LLMs.

Abstract

The application of large language models (LLMs) in the medical field has garnered significant attention, yet their reasoning capabilities in more specialized domains like anesthesiology remain underexplored. To bridge this gap, we introduce AnesSuite, the first comprehensive dataset suite specifically designed for anesthesiology reasoning in LLMs. The suite features AnesBench, an evaluation benchmark tailored to assess anesthesiology-related reasoning across three levels: factual retrieval (System 1), hybrid reasoning (System 1.x), and complex decision-making (System 2). Alongside this benchmark, the suite includes three training datasets that provide an infrastructure for continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning with verifiable rewards (RLVR). Leveraging this suite, we develop Morpheus, the first baseline model collection for anesthesiology reasoning. Despite undergoing limited training with SFT and group relative policy optimization (GRPO), Morpheus demonstrates substantial performance improvements, rivaling the performance of larger-scale models. Furthermore, through comprehensive evaluations and experiments, we analyze the key factors influencing anesthesiology reasoning performance, including model characteristics, training strategies and training data. Both AnesSuite and Morpheus will be open-sourced at https://github.com/MiliLab/AnesSuite.

Figures (13)

• Figure 1: Overview of AnesSuite. AnesSuite is composed of four components: AnesBench, a cross-lingual structured benchmark; AnesCorpus, a collection of anesthesiology documents; AnesQA, a question-answering dataset derived from domain literature; and AnesR1, a dataset featuring verifiable anesthesiology questions with chain-of-thought annotations. Leveraging this suite, we developed Morpheus, the first collection of reasoning LLMs for anesthesiology. Subsequent ablation and experiments identified key factors influencing reasoning performance in this specialized domain.
• Figure 1: Overview of datasets in AnesSuite.
• Figure 2: Number of choices in AnesBench and AnesR1.
• Figure 3: Distribution of AnesQA question type.
• Figure 4: Data length distribution for AnesBench, AnesQA, and AnesR1.