IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials

TL;DR

This study evaluates how large language models and biomedical PLMs perform safe and faithful natural language inference over breast cancer clinical trial reports (NLI4CT). By employing Retrieval-Augmented Generation with Tree of Thoughts and Chain-of-Thought prompting, the authors compare Gemini Pro, GPT-3.5, Flan-T5, and domain-specific PLMs under zero-shot and fine-tuned settings. They report F1=$0.69$, consistency=$0.71$, and faithfulness=$0.90$ for Gemini Pro on the official test, with ToT and CoT prompting providing substantial gains over baseline prompts, particularly for numerical reasoning where GPT-3.5 struggles. The work emphasizes instruction engineering and multi-turn reasoning to improve entailment/contradiction judgments across CTR sections, and provides open-source code to support reproducibility. Overall, the findings demonstrate that carefully crafted prompts and reasoning frameworks can enable robust, faithful biomedical NLI, informing safe evidence retrieval from clinical trial literature.

Abstract

Large Language models (LLMs) have demonstrated state-of-the-art performance in various natural language processing (NLP) tasks across multiple domains, yet they are prone to shortcut learning and factual inconsistencies. This research investigates LLMs' robustness, consistency, and faithful reasoning when performing Natural Language Inference (NLI) on breast cancer Clinical Trial Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We examine the reasoning capabilities of LLMs and their adeptness at logical problem-solving. A comparative analysis is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro under zero-shot settings using Retrieval-Augmented Generation (RAG) framework, integrating various reasoning chains. The evaluation yields an F1 score of 0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test dataset.

Figures (10)

• Figure 1: Examples of the dataset used in the NLI4CT task. Statement 1 compares the Intervention section from two different clinical trial reports, while statement 2 is based on the Adverse Events section of the first clinical trial report. The evaluation of the first statement requires textual inference skills, while the second requires numerical inference skills.
• Figure 2: An overview of the proposed system architecture used for the NLI4CT Task
• Figure 3: Instruction template for CoT prompting
• Figure 4: Prompt for Tree of Thought reasoning
• Figure 5: Final Instruction Template