Automated Benchmark Generation from Domain Guidelines Informed by Bloom's Taxonomy

TL;DR

BloomQA provides a domain-grounded benchmark design that circumvents reliance on exam banks by automatically extracting actionable practices from guidelines, turning them into violation-based scenarios, and expanding them into Bloom-enriched MCQs and multi-turn dialogues. The framework yields three large-scale datasets (Teach-QA, Diet-QA, CareGiving-QA) and demonstrates robust psychometric properties, including item discrimination and Bloom-level differentiation, across domains. It additionally shows that dialogue data can be used to finetune models for improved domain grounding, particularly in higher-order reasoning tasks. The approach offers a scalable, principled pathway to evaluate contextualized reasoning in real-world, practice-based settings.

Abstract

Open-ended question answering (QA) evaluates a model's ability to perform contextualized reasoning beyond factual recall. This challenge is especially acute in practice-based domains, where knowledge is procedural and grounded in professional judgment, while most existing LLM benchmarks depend on pre-existing human exam datasets that are often unavailable in such settings. We introduce a framework for automated benchmark generation from expert-authored guidelines informed by Bloom's Taxonomy. It converts expert practices into implicit violation-based scenarios and expands them into auto-graded multiple-choice questions (MCQs) and multi-turn dialogues across four cognitive levels, enabling deterministic, reproducible, and scalable evaluation. Applied to three applied domains: teaching, dietetics, and caregiving, we find differences between model and human-like reasoning: LLMs sometimes perform relatively better on higher-order reasoning (Analyze) but fail more frequently on lower-level items (Remember). We produce large-scale, psychometrically informed benchmarks that surface these non-intuitive model behaviors and enable evaluation of contextualized reasoning in real-world settings.

Figures (4)

• Figure 1: Bloom Hierarchical Progression Rates by Condition.
• Figure 2: Bloom Hierarchical Progression Rates by Condition and Direction.
• Figure 3: The figure shows how sensitive our conclusions are to the choice of $\Delta$ discrimination thresholds. Across a wide range of thresholds, the same pattern holds: Teaching practices exhibit substantially stronger discrimination than Diet or Caregiving, for both model-level separation ($\Delta_{\text{model}}$) and Bloom-level separation ($\Delta_{\text{bloom}}$). Bloom separation ($\Delta_{\text{bloom}}$): Teaching retains a large share of practices even at moderate $\Delta$ thresholds (0.20--0.30), whereas Diet and especially Caregiving drop sharply, indicating that Bloom-level distinctions are weakest in Caregiving. Model separation ($\Delta_{\text{model}}$): Teaching again retains the most practices across $\Delta$ values. Diet shows moderate discrimination, while Caregiving collapses first, indicating limited model-level spread. Overall, the ablation curves demonstrate that our findings are robust: Teaching produces higher-discrimination items, while Caregiving produces the lowest, regardless of where $\Delta$ thresholds are set.
• Figure 4: Scatterplots of observed versus expected correct counts under the GLMM for each Model$\times$Scenario cell. Points in green indicate cells well fit by the model (OK), while blue and red highlight cells performing better or worse than expected, respectively. The dashed diagonal represents perfect fit. Separate panels show the Teaching, Diet, and Caregiving domains.