The Radiation Oncology NLP Database

TL;DR

The paper introduces the Radiation Oncology NLP Database (ROND), the first dedicated NLP resource for radiation oncology, spanning logic reasoning, clinical text classification, NER, QA, summarization, and patient-clinician conversations to support domain-specific NLP research. It details construction of each task subset and presents an instruction-tuning pipeline that yields 20,160 pairs, including the CancerChat component, to demonstrate domain-focused instruction tuning with large language models. A CancerChat demonstration based on Falcon-7B with LoRA shows competitive performance against ChatGPT in a blind evaluation, highlighting the feasibility of domain-tuned models. Benchmark results across Bard, ChatGPT, and GPT-4 establish baselines and reveal GPT-4’s superior performance on most tasks, while also underscoring the need for further domain specialization to maximize clinical impact.

Abstract

We present the Radiation Oncology NLP Database (ROND), the first dedicated Natural Language Processing (NLP) dataset for radiation oncology, an important medical specialty that has received limited attention from the NLP community in the past. With the advent of Artificial General Intelligence (AGI), there is an increasing need for specialized datasets and benchmarks to facilitate research and development. ROND is specifically designed to address this gap in the domain of radiation oncology, a field that offers many opportunities for NLP exploration. It encompasses various NLP tasks including Logic Reasoning, Text Classification, Named Entity Recognition (NER), Question Answering (QA), Text Summarization, and Patient-Clinician Conversations, each with a distinct focus on radiation oncology concepts and application cases. In addition, we have developed an instruction-tuning dataset consisting of over 20k instruction pairs (based on ROND) and trained a large language model, CancerChat. This serves to demonstrate the potential of instruction-tuning large language models within a highly-specialized medical domain. The evaluation results in this study could serve as baseline results for future research. ROND aims to stimulate advancements in radiation oncology and clinical NLP by offering a platform for testing and improving algorithms and models in a domain-specific context. The ROND dataset is a joint effort of multiple U.S. health institutions. The data is available at https://github.com/zl-liu/Radiation-Oncology-NLP-Database.

Figures (7)

• Figure 1: Overview of the Radiation Oncology NLP Database.
• Figure 2: Illustration of the Clinical Text Classification dataset.
• Figure 3: A sample of the NER dataset.
• Figure 4: A sample multiple-choice question from the QA dataset.
• Figure 5: A detailed analysis of LLMs' performance on the medical physics board exam (RAPHEX) level questions.