Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Models

TL;DR

This study tackles public health surveillance of foodborne GI illness by leveraging online restaurant reviews and large language models. It introduces a novel annotation schema and conducts a comprehensive, multi-task evaluation of open-weight LLMs against a fine-tuned RoBERTa baseline for GI illness detection, symptom extraction, and food extraction using Yelp data and FoodEx2 mappings. Across extensive prompting experiments, LLMs achieve micro-F1 scores above 90% and often outperform smaller fine-tuned models, with robust performance across bias-focused checks. The work demonstrates the practical potential of prompt-based LLMs for extracting actionable health information from reviews while acknowledging data biases and attribution limitations that warrant cautious interpretation in surveillance applications.

Abstract

Foodborne gastrointestinal (GI) illness is a common cause of ill health in the UK. However, many cases do not interact with the healthcare system, posing significant challenges for traditional surveillance methods. The growth of publicly available online restaurant reviews and advancements in large language models (LLMs) present potential opportunities to extend disease surveillance by identifying public reports of GI illness. In this study, we introduce a novel annotation schema, developed with experts in GI illness, applied to the Yelp Open Dataset of reviews. Our annotations extend beyond binary disease detection, to include detailed extraction of information on symptoms and foods. We evaluate the performance of open-weight LLMs across these three tasks: GI illness detection, symptom extraction, and food extraction. We compare this performance to RoBERTa-based classification models fine-tuned specifically for these tasks. Our results show that using prompt-based approaches, LLMs achieve micro-F1 scores of over 90% for all three of our tasks. Using prompting alone, we achieve micro-F1 scores that exceed those of smaller fine-tuned models. We further demonstrate the robustness of LLMs in GI illness detection across three bias-focused experiments. Our results suggest that publicly available review text and LLMs offer substantial potential for public health surveillance of GI illness by enabling highly effective extraction of key information. While LLMs appear to exhibit minimal bias in processing, the inherent limitations of restaurant review data highlight the need for cautious interpretation of results.

Figures (4)

• Figure 1: Data from Food and You 2: Wave Six on the proportion of people eating catered food over the previous four weeks by annual household income Armstrong2023-px
• Figure 2: Data from the ONS on the mean weekly expenditure on catered food by annual household income decile ONS2024
• Figure 3: The protocol questions, and resulting manual annotations applied to support the GI classification task
• Figure 4: Examples of how raw text on food is processed