Reference-Free Evaluation of Taxonomies

TL;DR

This work tackles the challenge of evaluating taxonomies without reference labels by introducing two metrics: CSC, which measures robustness through the alignment of semantic and taxonomic structure, and NLIV, which gauges logical adequacy via NLI-based edge plausibility augmented with Hearst-pattern evidence. By connecting semantic proximity to taxonomic distance and by modeling plausible parent–child edges, the approach provides a reference-free toolkit that correlates with gold-standard F1 and can predict downstream hierarchical classification performance across diverse taxonomies. Extensive intrinsic and extrinsic evaluations across five taxonomies and multiple datasets demonstrate that CSC and NLIV offer strong, consistent signals, outperforming or complementing prior metrics like RaTE and SP, particularly in non-leaf mutation scenarios and when predicting real-world classification outcomes. The results support practical use in taxonomy design and evaluation, with limitations tied to the biases of underlying language models and the need for domain-aware validation.

Abstract

We introduce two reference-free metrics for quality evaluation of taxonomies in the absence of labels. The first metric evaluates robustness by calculating the correlation between semantic and taxonomic similarity, addressing error types not considered by existing metrics. The second uses Natural Language Inference to assess logical adequacy. Both metrics are tested on five taxonomies and are shown to correlate well with F1 against ground truth taxonomies. We further demonstrate that our metrics can predict downstream performance in hierarchical classification when used with label hierarchies.

Figures (14)

• Figure 1: Core ideas of our reference-free measures. We correlate semantic and taxonomy similarity to measure robustness and check parent-child edges to assess logical adequacy of the taxonomy.
• Figure 2: Toy taxonomy to illustrate different mutations that a robustness metric should be sensitive to. Leaf nodes are marked in blue, mutated nodes in yellow.
• Figure 3: The best and worst toy taxonomies according to the product of and NLIV-S in a 10k sample of the toy example space.
• Figure 4: Correlations between F1 and reference-free metrics, including the correlations when only mutating non-leaves. All correlations are statistically significant with $\alpha = 0.001$. We calculated F1 scores with and without class weights, assigning weights based on the number of descendants of a concept.
• Figure 5: Correlations between macro F1 on the downstream task and our reference-free metrics. All correlations are statistically significant with $\alpha = 0.001$.