SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning

TL;DR

This work introduces SylloBio-NLI, a framework for evaluating biomedical syllogistic reasoning in large language models by leveraging external ontologies (Reactome) to instantiate 28 domain-specific syllogistic schemes. It defines two NLI tasks (textual entailment and premise selection) and evaluates eight open-source LLMs in zero-shot and few-shot settings, reporting that zero-shot accuracy ranges from 70% on generalized modus ponens to 23% on disjunctive syllogism, with few-shot prompting boosting performance variably (Gemma +14%, Llama-3 +43%). The study also shows models are highly sensitive to lexical variants and domain content, indicating limited robustness for safe biomedical NLI. Overall, while few-shot prompts can elicit some syllogistic reasoning, current models still struggle to achieve reliable, domain-safe generalization in biomedical contexts, and the authors provide a reusable dataset and methodology to guide future work.

Abstract

Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support automatic evidence interpretation and scientific discovery. This paper presents SylloBio-NLI, a novel framework that leverages external ontologies to systematically instantiate diverse syllogistic arguments for biomedical NLI. We employ SylloBio-NLI to evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. Extensive experiments reveal that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism. At the same time, we found that few-shot prompting can boost the performance of different LLMs, including Gemma (+14%) and LLama-3 (+43%). However, a deeper analysis shows that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. Overall, our results indicate that, while in-context examples have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

Figures (15)

• Figure 1: End-to-end diagram of the proposed methodological framework, illustrating the generation of syllogistic arguments from domain-specific ontologies, parameterized input to LLMs, and evaluation tasks including textual inference and premise selection.
• Figure 2: Domain-specific pipeline for creating natural language instances of argument schemes with multiple templating. Steps include selecting a syllogistic schema (A), applying a domain-specific template (B), instantiating with predicates and entities (C), permuting premises (D), and prompting LLMs for evaluation (E).
• Figure 3: Comparative Analysis of accuracy, F1, and Faithfulness across two prompt types: ZS (left) and FS (right) for Task 1 for the four best models.
• Figure 4: Accuracy across two prompt types: ZS and FS, for Task 1 (top) and Task 2 (bottom). The lines connect the average accuracy for each of the seven syllogistic argument schemes, with green lines indicating an increase and red lines indicating a decrease. Gray boxplots display the median, Q1, Q3, and minimum and maximum values.
• Figure 5: Robustness to lexical variations. The graphs show the accuracy by model, syllogistic schema and lexical variations for ZS and FS respectively for Task 1 (top) and Task 2 (bottom).