Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using LLMs to label medical papers according to the CIViC evidence model

Markus Hisch, Xing David Wang

TL;DR

This work tackles the CIViC Evidence labeling problem, casting it as a multi-label sequence-classification task that assigns five evidence levels (A–E) to paper abstracts. It systematically compares encoder-only transformers (BERT/RoBERTa) with domain-specific variants (BiomedBERT, BioLinkBERT, BioMed-RoBERTa) and a long-context Biomed-RoBERTa-Long, plus OpenAI's GPT-4 in few-shot settings. The results show that domain-specific pretraining yields consistent gains over base models, with Biomed-RoBERTa-Long offering the best overall performance on CIViC Evidence; GPT-4 in few-shot underperforms relative to fine-tuned models but can approach baseline tf-idf performance under certain prompts. The findings support scalable automatic evidence labeling to power precision oncology knowledge bases and molecular tumor boards, facilitating more efficient literature curation.

Abstract

We introduce the sequence classification problem CIViC Evidence to the field of medical NLP. CIViC Evidence denotes the multi-label classification problem of assigning labels of clinical evidence to abstracts of scientific papers which have examined various combinations of genomic variants, cancer types, and treatment approaches. We approach CIViC Evidence using different language models: We fine-tune pretrained checkpoints of BERT and RoBERTa on the CIViC Evidence dataset and challenge their performance with models of the same architecture which have been pretrained on domain-specific text. In this context, we find that BiomedBERT and BioLinkBERT can outperform BERT on CIViC Evidence (+0.8% and +0.9% absolute improvement in class-support weighted F1 score). All transformer-based models show a clear performance edge when compared to a logistic regression trained on bigram tf-idf scores (+1.5 - 2.7% improved F1 score). We compare the aforementioned BERT-like models to OpenAI's GPT-4 in a few-shot setting (on a small subset of our original test dataset), demonstrating that, without additional prompt-engineering or fine-tuning, GPT-4 performs worse on CIViC Evidence than our six fine-tuned models (66.1% weighted F1 score compared to 71.8% for the best fine-tuned model). However, performance gets reasonably close to the benchmark of a logistic regression model trained on bigram tf-idf scores (67.7% weighted F1 score).

Using LLMs to label medical papers according to the CIViC evidence model

TL;DR

This work tackles the CIViC Evidence labeling problem, casting it as a multi-label sequence-classification task that assigns five evidence levels (A–E) to paper abstracts. It systematically compares encoder-only transformers (BERT/RoBERTa) with domain-specific variants (BiomedBERT, BioLinkBERT, BioMed-RoBERTa) and a long-context Biomed-RoBERTa-Long, plus OpenAI's GPT-4 in few-shot settings. The results show that domain-specific pretraining yields consistent gains over base models, with Biomed-RoBERTa-Long offering the best overall performance on CIViC Evidence; GPT-4 in few-shot underperforms relative to fine-tuned models but can approach baseline tf-idf performance under certain prompts. The findings support scalable automatic evidence labeling to power precision oncology knowledge bases and molecular tumor boards, facilitating more efficient literature curation.

Abstract

We introduce the sequence classification problem CIViC Evidence to the field of medical NLP. CIViC Evidence denotes the multi-label classification problem of assigning labels of clinical evidence to abstracts of scientific papers which have examined various combinations of genomic variants, cancer types, and treatment approaches. We approach CIViC Evidence using different language models: We fine-tune pretrained checkpoints of BERT and RoBERTa on the CIViC Evidence dataset and challenge their performance with models of the same architecture which have been pretrained on domain-specific text. In this context, we find that BiomedBERT and BioLinkBERT can outperform BERT on CIViC Evidence (+0.8% and +0.9% absolute improvement in class-support weighted F1 score). All transformer-based models show a clear performance edge when compared to a logistic regression trained on bigram tf-idf scores (+1.5 - 2.7% improved F1 score). We compare the aforementioned BERT-like models to OpenAI's GPT-4 in a few-shot setting (on a small subset of our original test dataset), demonstrating that, without additional prompt-engineering or fine-tuning, GPT-4 performs worse on CIViC Evidence than our six fine-tuned models (66.1% weighted F1 score compared to 71.8% for the best fine-tuned model). However, performance gets reasonably close to the benchmark of a logistic regression model trained on bigram tf-idf scores (67.7% weighted F1 score).
Paper Structure (22 sections, 5 equations, 12 figures, 11 tables)

This paper contains 22 sections, 5 equations, 12 figures, 11 tables.

Table of Contents

  1. Introduction
  2. The sequence classification problem CIViC Evidence
  3. Research questions
  4. Main contributions
  5. Background
  6. The role of automatic evidence classification in clinical practice
  7. Natural Language Processing using Transformers
  8. Task-specific prediction heads: Pretraining & Fine-tuning
  9. Few-shot Learning
  10. Explainable AI (XAI) by means of Integrated Gradients
  11. Related work
  12. Methodology
  13. The Dataset CIViC Evidence
  14. Machine Learning Models
  15. Evaluation setup
  16. ...and 7 more sections

Figures (12)

  • Figure 1: PubMed keyword search matches for Pancreatic Cancer and various molecular profiles
  • Figure 2: PubMed keyword search matches for Pancreatic Cancer and BRCA1 Mutation over time
  • Figure 3: Possible architecture of encoder-only and decoder-only transformers
  • Figure 4: Evidence level distribution given two distinct disease names and molecular profiles
  • Figure 5: Predicting labels of clinical evidence from abstracts of medical publications. Shown abstract from wuasdf
  • ...and 7 more figures