Description Boosting for Zero-Shot Entity and Relation Classification

TL;DR

This work tackles the sensitivity of zero-shot entity and relation classification to the textual descriptions of unseen classes. It introduces UDEBO, an unsupervised framework that generates diverse description variations, ranks them via entropy, and optionally ensembles predictions across variants to boost zero-shot performance. Empirical evaluations on OntoNotes, MedMentions, FewRel, and WikiZS show that UDEBO substantially improves Macro F1 over strong baselines, achieving new state-of-the-art results in several settings. The paper also analyzes the reliability of entropy-based description ranking, highlights limitations, discusses large language models, and provides open-source code and data to promote reproducibility and further study. Overall, the approach offers a practical pathway to improve zero-shot NER and RE in low-labeled-data regimes by leveraging automatically generated description variations and ensemble techniques.

Abstract

Zero-shot entity and relation classification models leverage available external information of unseen classes -- e.g., textual descriptions -- to annotate input text data. Thanks to the minimum data requirement, Zero-Shot Learning (ZSL) methods have high value in practice, especially in applications where labeled data is scarce. Even though recent research in ZSL has demonstrated significant results, our analysis reveals that those methods are sensitive to provided textual descriptions of entities (or relations). Even a minor modification of descriptions can lead to a change in the decision boundary between entity (or relation) classes. In this paper, we formally define the problem of identifying effective descriptions for zero shot inference. We propose a strategy for generating variations of an initial description, a heuristic for ranking them and an ensemble method capable of boosting the predictions of zero-shot models through description enhancement. Empirical results on four different entity and relation classification datasets show that our proposed method outperform existing approaches and achieve new SOTA results on these datasets under the ZSL settings. The source code of the proposed solutions and the evaluation framework are open-sourced.

Figures (5)

• Figure 1: A small modification of the Company class description results in different entity predictions.
• Figure 2: The figure shows the distributions of Macro F1 Score values on the test split of the OntoNotesZS dataset for each class, using the strategies described in Section \ref{['sec:desc_alter_strategies']} to generate 100 description variations for each class.
• Figure 3: The figure shows the distributions of Macro F1 Score values on the test split of the MedMentions dataset for each class, using the strategies described in Section \ref{['sec:desc_alter_strategies']} to generate 100 description variations for each class.
• Figure 4: Analysis of the correlation between entropy and Macro F1 Score on unseen classes on the OntoNotesZS test split. Entropy can be calculated without the need for labeled data, therefore, if a correlation exists it can be used as an unsupervised heuristic to select descriptions that improve model performance.
• Figure 5: Analysis of the correlation between entropy and Macro F1 Score on unseen classes on the MedmentionsZS test split.