From Memorization to Generalization: Fine-Tuning Large Language Models for Biomedical Term-to-Identifier Normalization

TL;DR

A predictive framework is provided for when fine-tuning enhances factual recall versus when it fails due to sparse or non-lexicalized identifiers due to sparse or non-lexicalized identifiers.

Abstract

Effective biomedical data integration depends on automated term normalization, the mapping of natural language biomedical terms to standardized identifiers. This linking of terms to identifiers is essential for semantic interoperability. Large language models (LLMs) show promise for this task but perform unevenly across terminologies. We evaluated both memorization (training-term performance) and generalization (validation-term performance) across multiple biomedical ontologies. Fine-tuning Llama 3.1 8B revealed marked differences by terminology. GO mappings showed strong memorization gains (up to 77% improvement in term-to-identifier accuracy), whereas HPO showed minimal improvement. Generalization occurred only for protein-gene (GENE) mappings (13.9% gain), while fine-tuning for HPO and GO yielded negligible transfer. Baseline accuracy varied by model scale, with GPT-4o outperforming both Llama variants for all terminologies. Embedding analyses showed tight semantic alignment between gene symbols and protein names but weak alignment between terms and identifiers for GO or HPO, consistent with limited lexicalization. Fine-tuning success depended on two interacting factors: identifier popularity and lexicalization. Popular identifiers were more likely encountered during pretraining, enhancing memorization. Lexicalized identifiers, such as gene symbols, enabled semantic generalization. By contrast, arbitrary identifiers in GO and HPO constrained models to rote learning. These findings provide a predictive framework for when fine-tuning enhances factual recall versus when it fails due to sparse or non-lexicalized identifiers.

Figures (5)

• Figure 1: Overview of Methods. Phase 1: frequency-balanced train and validation test sets were created from three biomedical terminologies. Phase 2: Fine-tuned models were created for each terminology and performance compared to baseline models. Phase 3: Gains from fine-tuning were attributed to either memorization or generalization. Effects of scale, popularity, directionality, and lexicalization were assessed.
• Figure 2: Bidirectional term mapping performance comparison across biomedical terminologies and model scales. Overall, Term$\rightarrow$identifier mappings perform better than identifier$\rightarrow$term, with Protein/Gene achieving the highest accuracy, GO moderate, and HPO the poorest across model scales. Performance evaluated on combined training and validation sets.
• Figure 3: Sankey diagrams illustrating fine-tuning outcomes across HPO, GO, and GENE models. Color flows visualize how terms transition after fine-tuning: blue = correct, orange = gainers (newly correct), red = incorrect, and green = degraded. Panel A (HPO) shows dominance of red flows, reflecting the long tail of low-popularity identifiers that remain unmapped after fine-tuning. Panel B (GO) exhibits strong orange flows, consistent with moderate popularity and effective memorization of trained terms. Panel C (GENE) displays extensive blue and orange flows, indicating high-popularity, lexicalized identifiers that support both memorization and generalization to unseen terms. Green (degradation) flows are visible mainly in the GENE model, marking the upper limit of representational saturation.
• Figure 4: Identifier frequency and semantic alignment patterns across biomedical terminologies based on the 200 trained terms only. (A) Log-log rank-frequency plot of identifiers in PubMed Central showing GENE symbols (red circles) in the distribution head, GO identifiers (blue circles) in the body, and HPO identifiers (green circles) in the long tail. (B) PCA projection of Llama 3.1 8B embeddings showing overlapping clusters for GENE terms (red circles) and identifiers (red squares) versus clear separation between terms (circles) and identifiers (squares) for GO (blue) and HPO (green).
• Figure 5: Semantic alignment between terms and identifiers across biomedical terminologies based on the trained terms only. (A) Row-wise vs. non–row-wise cosine similarity reveals semantic alignment only for GENE term--identifier pairs (red bars). Bars show mean cosine similarity ($\pm$SD) between each term and its own identifier (solid color) versus the same term paired with all other identifiers in the same terminology (lighter shade). (B) Euclidean distances between terms and identifier in PCA space (4096 $\rightarrow$ 2 dimensions). GENE term–identifier pairs (red box) are close to each other, unlike GO and HPO.