Better Generalizing to Unseen Concepts: An Evaluation Framework and An LLM-Based Auto-Labeled Pipeline for Biomedical Concept Recognition

TL;DR

This paper tackles the challenge of recognizing unseen biomedical concepts by reframing Biomedical Concept Recognition as Mention-agnostic (MA-BCR) and introducing an evaluation framework grounded in hierarchical concept indices. It proposes an LLM-based auto-labeling pipeline (ALD) to expand concept coverage and provide structural signals, while carefully assessing how such noisy data affects generalization to unseen concepts. Through three index types (OSI, SSI, OSSI) and two new metrics, Unseen Recall-oriented Closeness ($U$-RC) and Unseen Candidate-set Size ($U$-CS), the authors demonstrate that ALD improves hierarchy-aware generalization even though exact-match performance lags behind manual labels. The combination of hierarchical indexing and scalable ALD offers a practical pathway to broader concept coverage and stronger unseen-concept recognition, with implications for building scalable biomedical knowledge bases and downstream decision-support systems. Limitations include remaining gaps in HOIP/HPO coverage, varying LLM capabilities across stages, and the need for improved automatic evaluation alignment with human judgments.

Abstract

Generalization to unseen concepts is a central challenge due to the scarcity of human annotations in Mention-agnostic Biomedical Concept Recognition (MA-BCR). This work makes two key contributions to systematically address this issue. First, we propose an evaluation framework built on hierarchical concept indices and novel metrics to measure generalization. Second, we explore LLM-based Auto-Labeled Data (ALD) as a scalable resource, creating a task-specific pipeline for its generation. Our research unequivocally shows that while LLM-generated ALD cannot fully substitute for manual annotations, it is a valuable resource for improving generalization, successfully providing models with the broader coverage and structural knowledge needed to approach recognizing unseen concepts. Code and datasets are available at https://github.com/bio-ie-tool/hi-ald.

Figures (11)

• Figure 1: Overview of the LLM-based Auto-labeling Pipeline. Given an input passage, the pipeline begins by generating intermediate claims (Arrow 1), followed by generating candidate concept names from the claims (Arrow 2). The resulting names are matched to ontology terms by comparing the representations (encoded by SapBERT) of generated and ontological names, forming a preliminary list of concept candidates, shown in the blue-shaded table. Then concept classification, relabeling, and guideline-based filtering steps are applied to get the final annotations (highlighted in bold). The instance will be taken as a training instance if its quality meets the requirement.
• Figure 2: Overview of hierarchical search index construction. Ontology concepts are represented as nodes in a graph (yellow), where edges reflect either ontological relations (blue), semantic similarities (black), or both. A graph partitioning algorithm (e.g., Louvain) is then applied to recursively divide the graph into nested subgraphs. In the illustrated example, the initial graph is first partitioned into two coarse-level clusters (labeled 0 and 1), which are further subdivided into finer clusters (e.g., 0-0, 0-1, 1-0, 1-1). Based on the partitioned structure, a hierarchical label tree is constructed: internal nodes (gray) represent clusters, while leaf nodes (yellow) correspond to individual ontology concepts. Each concept is assigned an index reflecting its position in the tree (e.g., concept A is labeled 0-0-0, indicating its membership in cluster 0, then 0-0, and finally its position).
• Figure 3: Upstream metrics vs. downstream reranker F1. Each panel plots one recognizer metric against the reranker F1 across eight recognizers trained with different data volumes. $\rho$ denotes Spearman’s rank correlation between the upstream metric and reranker F1.
• Figure 4: Distribution of false positive outcomes in the LLM-based auto-labeling pipeline. "Missing gold" denotes predictions supported by the passage but absent from dataset-provided gold annotations; the remaining bars correspond to five error types.
• Figure 5: The prompts we used for PCC stage for HoIP concept generation.