Comparative Analysis of LoRA-Adapted Embedding Models for Clinical Cardiology Text Representation

TL;DR

The paper evaluates ten transformer architectures, fine-tuned with Low-Rank Adaptation (LoRA) on ~150k cardiology textbook pairs, to compare encoder-only versus decoder-style embeddings for domain-specific text representation. It introduces a cardiology-focused semantic separation metric and a rigorous evaluation framework, revealing that encoder-only models, especially BioLinkBERT, achieve the highest domain discrimination (up to 0.510) with substantially lower computational requirements than large decoders. LoRA fine-tuning dramatically improves performance over zero-shot baselines (median gain ~700%), transforming near-random embeddings into clinically useful tools in about 2 hours of A100 compute. The findings underscore the importance of domain-relevant pre-training and architectural choice over mere model scale, offering practical deployment guidance and enabling scalable creation of domain-specialized embeddings with minimal cost.

Abstract

Domain-specific text embeddings are critical for clinical natural language processing, yet systematic comparisons across model architectures remain limited. This study evaluates ten transformer-based embedding models adapted for cardiology through Low-Rank Adaptation (LoRA) fine-tuning on 106,535 cardiology text pairs derived from authoritative medical textbooks. Results demonstrate that encoder-only architectures, particularly BioLinkBERT, achieve superior domain-specific performance (separation score: 0.510) compared to larger decoder-based models, while requiring significantly fewer computational resources. The findings challenge the assumption that larger language models necessarily produce better domain-specific embeddings and provide practical guidance for clinical NLP system development. All models, training code, and evaluation datasets are publicly available to support reproducible research in medical informatics.

Figures (13)

• Figure 1: Which architectures achieve the best cardiology semantic discrimination? Encoder-only architectures (blue bars) dominate the top positions, with BioLinkBERT achieving the highest separation score (0.510). Eight of ten models exceed the clinical utility threshold of 0.3 (dashed line), with encoder-only models occupying four of the top five positions.
• Figure 2: What is the trade-off between semantic performance and inference speed? Three distinct clusters emerge: high-performance models (BioLinkBERT, Gemma-2-2B achieving 0.46--0.51 separation with moderate throughput), balanced models (MPNet-base, BGE-large-v1.5, Qwen2.5-0.5B with 0.31--0.39 separation and 88--229 emb/sec), and high-efficiency models (BGE-small-v1.5 delivering 467 emb/sec at 0.25 separation). Marker size is inversely proportional to GPU memory consumption (larger markers = less memory). BioLinkBERT's position in the upper-right quadrant demonstrates that encoder-only architectures can simultaneously achieve superior domain adaptation and acceptable inference speed, unlike decoder-style models (Qwen3-4B) which sacrifice efficiency for performance.
• Figure 3: Does parameter count predict GPU memory requirements? No clear relationship exists: parameter count alone is insufficient to predict memory usage. Models are colored by separation score performance (green = high, orange = medium, red = low). Embedding dimension plays a more critical role than parameter count in determining memory footprint.
• Figure 4: How does embedding dimensionality affect computational efficiency? Dimensionality inversely correlates with throughput (left panel) and directly correlates with memory consumption (right panel). Higher-dimensional embeddings require substantially more GPU memory while delivering lower inference speed. Notably, BioLinkBERT (1024-dim) achieves 0.51 separation while Qwen3-4B (2560-dim) achieves only 0.45, demonstrating that specialized encoder architectures with moderate dimensions outperform larger, higher-dimensional decoder models.
• Figure 5: Does LoRA fine-tuning fundamentally transform domain-specific performance? Yes. Gray bars show zero-shot (unmodified base model) separation scores, while blue bars show LoRA-adapted performance. Most unmodified models achieve near-zero separation scores (0.03--0.06), indicating inability to discriminate between related and unrelated cardiology concepts without domain-specific fine-tuning. LoRA adaptation delivers 2--15$\times$ performance improvements across 9 out of 10 models, with BioLinkBERT showing the most dramatic transformation (0.033 to 0.510, +1452%). Red arrows indicate improvement magnitude for top performers. Only Jina-v2 degrades with LoRA (-380%), suggesting architectural incompatibility. This demonstrates that general-purpose embedding models are fundamentally insufficient for specialized medical domains; domain-specific fine-tuning via LoRA is essential for clinical utility.