Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging

TL;DR

The paper tackles reliable extraction of pathologic $pTNM$ cancer stage from free-text pathology reports using prompting-based LLMs. It introduces Ensemble-Reasoning (EnsReas), a two-stage approach that first leverages zero-shot chain-of-thought with self-consistency to generate multiple reasonings, then identifies consistent vs. inconsistent predictions and applies a panel-style re-evaluation to refine the inconsistent cases. Using Med42-70B on a real TCGA breast cancer pathology dataset, EnsReas outperforms zero-shot, ZS-CoT, and ZS-CoT-SC baselines and significantly reduces output entropy, indicating higher consistency. The method is cost-efficient since re-evaluation targets only inconsistent predictions and does not rely on external knowledge, supporting more trustworthy deployment of clinical LLMs for cancer staging and potentially other constrained medical NLP tasks.

Abstract

Advances in large language models (LLMs) have encouraged their adoption in the healthcare domain where vital clinical information is often contained in unstructured notes. Cancer staging status is available in clinical reports, but it requires natural language processing to extract the status from the unstructured text. With the advance in clinical-oriented LLMs, it is promising to extract such status without extensive efforts in training the algorithms. Prompting approaches of the pre-trained LLMs that elicit a model's reasoning process, such as chain-of-thought, may help to improve the trustworthiness of the generated responses. Using self-consistency further improves model performance, but often results in inconsistent generations across the multiple reasoning paths. In this study, we propose an ensemble reasoning approach with the aim of improving the consistency of the model generations. Using an open access clinical large language model to determine the pathologic cancer stage from real-world pathology reports, we show that the ensemble reasoning approach is able to improve both the consistency and performance of the LLM in determining cancer stage, thereby demonstrating the potential to use these models in clinical or other domains where reliability and trustworthiness are critical.

Figures (1)

• Figure 1: Analysis of reasoning type between ZS-CoT-ZS and EnsReas.