StructEval: Deepen and Broaden Large Language Model Assessment via Structured Evaluation

TL;DR

StructEval tackles the shortcomings of single‑item LLM evaluations by introducing a structured, multi‑level, and multi‑node assessment framework. Grounded in Bloom’s Taxonomy and Concept Mapping, it deepens evaluation around a test objective and broadens it via knowledge‑graph–driven concept exploration. The framework automates large‑scale benchmark construction, demonstrates robustness to data contamination, and yields more consistent model rankings than prior augmentation methods. Its modular design enables dynamic, customizable benchmarks and offers guidance for principled future evaluation protocols in LLM research.

Abstract

Evaluation is the baton for the development of large language models. Current evaluations typically employ a single-item assessment paradigm for each atomic test objective, which struggles to discern whether a model genuinely possesses the required capabilities or merely memorizes/guesses the answers to specific questions. To this end, we propose a novel evaluation framework referred to as StructEval. Starting from an atomic test objective, StructEval deepens and broadens the evaluation by conducting a structured assessment across multiple cognitive levels and critical concepts, and therefore offers a comprehensive, robust and consistent evaluation for LLMs. Experiments on three widely-used benchmarks demonstrate that StructEval serves as a reliable tool for resisting the risk of data contamination and reducing the interference of potential biases, thereby providing more reliable and consistent conclusions regarding model capabilities. Our framework also sheds light on the design of future principled and trustworthy LLM evaluation protocols.

Figures (5)

• Figure 1: The illustrations for previous single-item assessment and our structured evaluation paradigm.
• Figure 2: The illustration of StructEval framework, which consists of two modules. The first module aims to evaluate the model's ability on test objective across multiple cognitive levels in Bloom's Taxonomy. The second module aims to evaluate the model's understanding of relevant critical concepts based on knowledge graph.
• Figure 3: The comparison of overall rank consistency for each method. StructEval substantially outperforms original benchmark and all augmentation-based strategies as number of sampled subjects $K$ changes.
• Figure 4: In addition to expanding on existing benchmarks, StructEval can also function as a customized benchmark construction framework. It is capable of automated data construction and evaluation tailored to assessment objectives of any granularity.
• Figure 5: The annotation guidelines for our human evaluation.