Towards Scalable and Cross-Lingual Specialist Language Models for Oncology
Morteza Rohanian, Tarun Mehra, Nicola Miglino, Farhad Nooralahzadeh, Michael Krauthammer, Andreas Wicki
TL;DR
The paper addresses the difficulty of extracting reliable, actionable insights from unstructured oncology data using general LLMs that lack domain-specific reasoning. It introduces an oncology-specialized NLP framework that fuses instruction tuning, retrieval-augmented generation (RAG), and graph-based knowledge integration on lightweight LLaMA models, with an objective for instruction tuning defined as $L_{tuning} = -\\frac{1}{N} \\sum_{i=1}^N \\log P_\\theta(y_i | x_i, instruction)$. The pipeline grounds outputs in external knowledge via FAISS-based retrieval from sources like MIMIC-IV and German discharge reports and a UMLS/SNOMED-CT/ICD-10 knowledge graph, enabling robust NER, relation extraction, TNM staging, and treatment-prediction across English and German data. Empirical results show consistent gains across tasks, with notable cross-lingual improvements when using as few as 100–400 German instructions, and demonstrate that smaller models can achieve competitive performance when paired with retrieval and graph grounding. The work offers a scalable, resource-efficient path for multilingual oncology NLP applicable to resource-constrained healthcare settings and points to future multimodal extensions and broader language coverage.
Abstract
Clinical oncology generates vast, unstructured data that often contain inconsistencies, missing information, and ambiguities, making it difficult to extract reliable insights for data-driven decision-making. General-purpose large language models (LLMs) struggle with these challenges due to their lack of domain-specific reasoning, including specialized clinical terminology, context-dependent interpretations, and multi-modal data integration. We address these issues with an oncology-specialized, efficient, and adaptable NLP framework that combines instruction tuning, retrieval-augmented generation (RAG), and graph-based knowledge integration. Our lightweight models prove effective at oncology-specific tasks, such as named entity recognition (e.g., identifying cancer diagnoses), entity linking (e.g., linking entities to standardized ontologies), TNM staging, document classification (e.g., cancer subtype classification from pathology reports), and treatment response prediction. Our framework emphasizes adaptability and resource efficiency. We include minimal German instructions, collected at the University Hospital Zurich (USZ), to test whether small amounts of non-English language data can effectively transfer knowledge across languages. This approach mirrors our motivation for lightweight models, which balance strong performance with reduced computational costs, making them suitable for resource-limited healthcare settings. We validated our models on oncology datasets, demonstrating strong results in named entity recognition, relation extraction, and document classification.