Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TaxoAlign: Scholarly Taxonomy Generation Using Language Models

Avishek Lahiri, Yufang Hou, Debarshi Kumar Sanyal

TL;DR

This work introduces CS-TaxoBench, a large-scale benchmark of scholarly taxonomies derived from real survey papers, plus an 80-taxonomy conference set, to enable rigorous evaluation of taxonomy generation. It then presents TaxoAlign, a three-phase pipeline (Knowledge Slice Creation, Taxonomy Verbalization, Taxonomy Refinement) that leverages instruction-tuned LLMs to produce structure- and content-grounded scholarly taxonomies from reference papers. The authors define novel structural and semantic evaluation metrics, including Average Degree Score, Level-order Traversal, Node Soft Recall, and Node Entity Recall, and show TaxoAlign consistently outperforms baselines on automated and human assessments, including testing on conference papers. The work provides open-source code and data, highlighting the practical potential to accelerate literature reviews and knowledge synthesis while revealing avenues for extending taxonomy quality and coverage in future research.

Abstract

Taxonomies play a crucial role in helping researchers structure and navigate knowledge in a hierarchical manner. They also form an important part in the creation of comprehensive literature surveys. The existing approaches to automatic survey generation do not compare the structure of the generated surveys with those written by human experts. To address this gap, we present our own method for automated taxonomy creation that can bridge the gap between human-generated and automatically-created taxonomies. For this purpose, we create the CS-TaxoBench benchmark which consists of 460 taxonomies that have been extracted from human-written survey papers. We also include an additional test set of 80 taxonomies curated from conference survey papers. We propose TaxoAlign, a three-phase topic-based instruction-guided method for scholarly taxonomy generation. Additionally, we propose a stringent automated evaluation framework that measures the structural alignment and semantic coherence of automatically generated taxonomies in comparison to those created by human experts. We evaluate our method and various baselines on CS-TaxoBench, using both automated evaluation metrics and human evaluation studies. The results show that TaxoAlign consistently surpasses the baselines on nearly all metrics. The code and data can be found at https://github.com/AvishekLahiri/TaxoAlign.

TaxoAlign: Scholarly Taxonomy Generation Using Language Models

TL;DR

This work introduces CS-TaxoBench, a large-scale benchmark of scholarly taxonomies derived from real survey papers, plus an 80-taxonomy conference set, to enable rigorous evaluation of taxonomy generation. It then presents TaxoAlign, a three-phase pipeline (Knowledge Slice Creation, Taxonomy Verbalization, Taxonomy Refinement) that leverages instruction-tuned LLMs to produce structure- and content-grounded scholarly taxonomies from reference papers. The authors define novel structural and semantic evaluation metrics, including Average Degree Score, Level-order Traversal, Node Soft Recall, and Node Entity Recall, and show TaxoAlign consistently outperforms baselines on automated and human assessments, including testing on conference papers. The work provides open-source code and data, highlighting the practical potential to accelerate literature reviews and knowledge synthesis while revealing avenues for extending taxonomy quality and coverage in future research.

Abstract

Taxonomies play a crucial role in helping researchers structure and navigate knowledge in a hierarchical manner. They also form an important part in the creation of comprehensive literature surveys. The existing approaches to automatic survey generation do not compare the structure of the generated surveys with those written by human experts. To address this gap, we present our own method for automated taxonomy creation that can bridge the gap between human-generated and automatically-created taxonomies. For this purpose, we create the CS-TaxoBench benchmark which consists of 460 taxonomies that have been extracted from human-written survey papers. We also include an additional test set of 80 taxonomies curated from conference survey papers. We propose TaxoAlign, a three-phase topic-based instruction-guided method for scholarly taxonomy generation. Additionally, we propose a stringent automated evaluation framework that measures the structural alignment and semantic coherence of automatically generated taxonomies in comparison to those created by human experts. We evaluate our method and various baselines on CS-TaxoBench, using both automated evaluation metrics and human evaluation studies. The results show that TaxoAlign consistently surpasses the baselines on nearly all metrics. The code and data can be found at https://github.com/AvishekLahiri/TaxoAlign.
Paper Structure (42 sections, 4 equations, 7 figures, 6 tables)

This paper contains 42 sections, 4 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Taxonomy Construction.
  4. Scientific Survey Generation and Knowledge Synthesis.
  5. CS-TaxoBench
  6. Overview
  7. Desiderata
  8. Automated Taxonomy Extraction
  9. Dataset Statistics
  10. Manual Annotation
  11. Additional Test Set: Survey Papers in Conferences
  12. TaxoAlign
  13. Task Formulation
  14. Knowledge Slice Creation
  15. Taxonomy Verbalization
  16. ...and 27 more sections

Figures (7)

  • Figure 1: Schematic representation of TaxoAlign: (1) Knowledge Slice Creation (2) Taxonomy Verbalization (3) Taxonomy Refinement
  • Figure 2: A comparison of a gold standard taxonomy tree and a generated taxonomy tree using TaxoAlign for the topic "Human Image Generation: A Comprehensive Survey". The generated taxonomy shown here uses Mistral-7B-Instruct-v0.3 for creation of knowledge slices and Llama-3.1-Tülu-3-8B for the taxonomy verbalization component and GPT-4o-mini for refining the generated taxonomy.
  • Figure 3: An overview of the pipeline for the curation of our dataset.
  • Figure 4: An overview of the proposed TaxoAlign pipeline.
  • Figure 5: A comparison of a gold standard taxonomy tree and a generated taxonomy tree using AutoSurvey.
  • ...and 2 more figures