Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LITE: LLM-Impelled efficient Taxonomy Evaluation

Lin Zhang, Zhouhong Gu, Suhang Zheng, Tao Wang, Tianyu Li, Hongwei Feng, Yanghua Xiao

TL;DR

The paper tackles the challenge of scalable, task-aligned taxonomy evaluation by introducing LiTe, an LLM-driven framework that evaluates taxonomies through a top-down subtree sampling strategy, standardized JSON inputs, cross-validation, and a penalty mechanism to stabilize scores. It defines four metrics—Single Concept Accuracy (SCA), Hierarchy Relationship Rationality (HRR), Hierarchy Relationship Exclusivity (HRE), and Hierarchy Relationship Independence (HRI)—to capture concept clarity, hierarchical validity, exclusivity, and independence. Empirical results on MAG-CS and Ali-Taxo demonstrate strong alignment with human judgments, robust detection of semantic, logical, and structural flaws, and clear guidance for taxonomy refinement, across diverse datasets and ablations. Overall, LiTe advances practical taxonomy quality assurance by leveraging LLMs while addressing efficiency, fairness, and consistency concerns, making it a viable tool for knowledge organization in real-world settings.

Abstract

This paper presents LITE, an LLM-based evaluation method designed for efficient and flexible assessment of taxonomy quality. To address challenges in large-scale taxonomy evaluation, such as efficiency, fairness, and consistency, LITE adopts a top-down hierarchical evaluation strategy, breaking down the taxonomy into manageable substructures and ensuring result reliability through cross-validation and standardized input formats. LITE also introduces a penalty mechanism to handle extreme cases and provides both quantitative performance analysis and qualitative insights by integrating evaluation metrics closely aligned with task objectives. Experimental results show that LITE demonstrates high reliability in complex evaluation tasks, effectively identifying semantic errors, logical contradictions, and structural flaws in taxonomies, while offering directions for improvement. Code is available at https://github.com/Zhang-l-i-n/TAXONOMY_DETECT .

LITE: LLM-Impelled efficient Taxonomy Evaluation

TL;DR

The paper tackles the challenge of scalable, task-aligned taxonomy evaluation by introducing LiTe, an LLM-driven framework that evaluates taxonomies through a top-down subtree sampling strategy, standardized JSON inputs, cross-validation, and a penalty mechanism to stabilize scores. It defines four metrics—Single Concept Accuracy (SCA), Hierarchy Relationship Rationality (HRR), Hierarchy Relationship Exclusivity (HRE), and Hierarchy Relationship Independence (HRI)—to capture concept clarity, hierarchical validity, exclusivity, and independence. Empirical results on MAG-CS and Ali-Taxo demonstrate strong alignment with human judgments, robust detection of semantic, logical, and structural flaws, and clear guidance for taxonomy refinement, across diverse datasets and ablations. Overall, LiTe advances practical taxonomy quality assurance by leveraging LLMs while addressing efficiency, fairness, and consistency concerns, making it a viable tool for knowledge organization in real-world settings.

Abstract

This paper presents LITE, an LLM-based evaluation method designed for efficient and flexible assessment of taxonomy quality. To address challenges in large-scale taxonomy evaluation, such as efficiency, fairness, and consistency, LITE adopts a top-down hierarchical evaluation strategy, breaking down the taxonomy into manageable substructures and ensuring result reliability through cross-validation and standardized input formats. LITE also introduces a penalty mechanism to handle extreme cases and provides both quantitative performance analysis and qualitative insights by integrating evaluation metrics closely aligned with task objectives. Experimental results show that LITE demonstrates high reliability in complex evaluation tasks, effectively identifying semantic errors, logical contradictions, and structural flaws in taxonomies, while offering directions for improvement. Code is available at https://github.com/Zhang-l-i-n/TAXONOMY_DETECT .

Paper Structure

This paper contains 42 sections, 3 equations, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Framework
  5. Step 1: Subtree Selection
  6. Step 2: Scoring
  7. Standardized Input Format
  8. Cross-Validation Techniques
  9. Step 3: Handling Extreme Subtrees
  10. Evaluation Metrics
  11. Single Concept Accuracy (SCA)
  12. Hierarchy Relationship Rationality (HRR)
  13. Hierarchy Relationship Exclusivity (HRE)
  14. Hierarchy Relationship Independence (HRI)
  15. Experiment
  16. ...and 27 more sections

Figures (2)

  • Figure 1: The trend of evaluation efficiency with subtree node count, where the solid line shows the success rate, and the dashed lines represent the average token cost for input (black), output (red), and input (green). The token cost axis is logarithmic, with higher values (lower position) indicating lower efficiency. The parameter K in formula \ref{['eq:1']} defines the subtree size, with K set to None indicating evaluation with the smallest subtree.
  • Figure 2: Evaluation prompt