Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Epidemic Information Extraction for Event-Based Surveillance using Large Language Models

Sergio Consoli, Peter Markov, Nikolaos I. Stilianakis, Lorenzo Bertolini, Antonio Puertas Gallardo, Mario Ceresa

TL;DR

The study investigates extracting structured epidemic information from unstructured sources (ProMED and DONs) using both open-source and commercial large language models (LLMs) and a traditional tool, EpiTator. It assesses single models and an Open-Ensemble approach across a gold-standard IDB subset, measuring precision, recall, and F1 for fields like virus, country, date, and case counts, with GPT-4-32k and GPT-4-FewShots delivering top performance. Open-source models such as Llama-2-70b-chat, Mistral-7b-openorca, and Zephyr-7b-alpha also show strong results, and their ensemble (Open-Ensemble) yields robust performance comparable to GPTs, enabling deployment on the full dataset (~70k documents). The work demonstrates that LLMs can enhance the accuracy and timeliness of epidemic modelling and forecasting, offering a scalable tool for improving early warning and response in future pandemics, while highlighting trade-offs between cost and deployment practicality.

Abstract

This paper presents a novel approach to epidemic surveillance, leveraging the power of Artificial Intelligence and Large Language Models (LLMs) for effective interpretation of unstructured big data sources, like the popular ProMED and WHO Disease Outbreak News. We explore several LLMs, evaluating their capabilities in extracting valuable epidemic information. We further enhance the capabilities of the LLMs using in-context learning, and test the performance of an ensemble model incorporating multiple open-source LLMs. The findings indicate that LLMs can significantly enhance the accuracy and timeliness of epidemic modelling and forecasting, offering a promising tool for managing future pandemic events.

Epidemic Information Extraction for Event-Based Surveillance using Large Language Models

TL;DR

The study investigates extracting structured epidemic information from unstructured sources (ProMED and DONs) using both open-source and commercial large language models (LLMs) and a traditional tool, EpiTator. It assesses single models and an Open-Ensemble approach across a gold-standard IDB subset, measuring precision, recall, and F1 for fields like virus, country, date, and case counts, with GPT-4-32k and GPT-4-FewShots delivering top performance. Open-source models such as Llama-2-70b-chat, Mistral-7b-openorca, and Zephyr-7b-alpha also show strong results, and their ensemble (Open-Ensemble) yields robust performance comparable to GPTs, enabling deployment on the full dataset (~70k documents). The work demonstrates that LLMs can enhance the accuracy and timeliness of epidemic modelling and forecasting, offering a scalable tool for improving early warning and response in future pandemics, while highlighting trade-offs between cost and deployment practicality.

Abstract

This paper presents a novel approach to epidemic surveillance, leveraging the power of Artificial Intelligence and Large Language Models (LLMs) for effective interpretation of unstructured big data sources, like the popular ProMED and WHO Disease Outbreak News. We explore several LLMs, evaluating their capabilities in extracting valuable epidemic information. We further enhance the capabilities of the LLMs using in-context learning, and test the performance of an ensemble model incorporating multiple open-source LLMs. The findings indicate that LLMs can significantly enhance the accuracy and timeliness of epidemic modelling and forecasting, offering a promising tool for managing future pandemic events.
Paper Structure (17 sections, 4 figures)

This paper contains 17 sections, 4 figures.

Table of Contents

  1. Introduction and background
  2. Data
  3. ProMED
  4. WHO Disease Outbreaks News
  5. Methods
  6. EpiTator
  7. Pythia-12b
  8. Mpt-30b-chat
  9. Llama-2-70b-chat
  10. Mistral-7b-openorca
  11. Zephyr-7b-alpha
  12. Gpt-3.5-turbo-16k
  13. Gpt-4-32k
  14. Gpt-4-FewShots
  15. Open-Ensemble
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Comparison of the models for the extraction of the pandemic name.
  • Figure 2: Comparison of the models for the extraction of the country name.
  • Figure 3: Comparison of the models for the extraction of the pandemic date.
  • Figure 4: Comparison of the models for the extraction of the number of cases engender by the pandemic.