Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkish

TL;DR

This work addresses the question of how to perform clinical relation extraction (RE) in a low-resource language by evaluating prompting-based large language models (LLMs) on English–Turkish clinical text. It introduces the first English–Turkish parallel RE dataset derived from i2b2/VA, and systematically compares in-context learning, chain-of-thought prompting, and fine-tuning baselines. A central contribution is Relation-Aware Retrieval (RAR), a contrastive-learning–based demonstration retriever that aligns sentence, entity, and relation semantics; when combined with Output Format–based CoT, it achieves state-of-the-art bilingual RE performance (English micro-F1 ≈0.918, Turkish ≈0.888–0.918 across settings). The findings highlight that high-quality demonstration retrieval and reasoning-aware prompts can bridge resource gaps in clinical NLP, enabling robust cross-lingual performance with limited bilingual data and reducing reliance on large language-model fine-tuning.

Abstract

The scarcity of annotated datasets for clinical information extraction in non-English languages hinders the evaluation of large language model (LLM)-based methods developed primarily in English. In this study, we present the first comprehensive bilingual evaluation of LLMs for the clinical Relation Extraction (RE) task in both English and Turkish. To facilitate this evaluation, we introduce the first English-Turkish parallel clinical RE dataset, derived and carefully curated from the 2010 i2b2/VA relation classification corpus. We systematically assess a diverse set of prompting strategies, including multiple in-context learning (ICL) and Chain-of-Thought (CoT) approaches, and compare their performance to fine-tuned baselines such as PURE. Furthermore, we propose Relation-Aware Retrieval (RAR), a novel in-context example selection method based on contrastive learning, that is specifically designed to capture both sentence-level and relation-level semantics. Our results show that prompting-based LLM approaches consistently outperform traditional fine-tuned models. Moreover, evaluations for English performed better than their Turkish counterparts across all evaluated LLMs and prompting techniques. Among ICL methods, RAR achieves the highest performance, with Gemini 1.5 Flash reaching a micro-F1 score of 0.906 in English and 0.888 in Turkish. Performance further improves to 0.918 F1 in English when RAR is combined with a structured reasoning prompt using the DeepSeek-V3 model. These findings highlight the importance of high-quality demonstration retrieval and underscore the potential of advanced retrieval and prompting techniques to bridge resource gaps in clinical natural language processing.

Figures (2)

• Figure 1: Comparison of different methods across different shot settings.
• Figure 2: Confusion matrices for English (left) and Turkish (right).