Small Language Models Can Use Nuanced Reasoning For Health Science Research Classification: A Microbial-Oncogenesis Case Study

TL;DR

The paper evaluates whether small language models (≤8B parameters) can perform nuanced biomedical literature screening, focusing on the HMTV/MMTV-like virus and breast cancer case. It compares zero-shot and in-context learning with programmatic prompt optimization against frontier LLMs, finding that Llama3 and Qwen2.5 can match or approach Gemini 2.5 Pro on key filtering tasks, especially for the RELEVANT classification. Perturbation analyses reveal that SLM decisions rest on legitimate biomedical cues but are susceptible to spurious textual artifacts, underscoring the need for interpretability and human oversight in high-stakes workflows. The study provides a public expert-validated dataset and argues for using SLMs as cost-efficient first-pass filters in AI co-scientist pipelines to accelerate scalable discovery of microbe–cancer associations.

Abstract

Artificially intelligent (AI) co-scientists must be able to sift through research literature cost-efficiently while applying nuanced scientific reasoning. We evaluate Small Language Models (SLMs, <= 8B parameters) for classifying medical research papers. Using literature on the oncogenic potential of HMTV/MMTV-like viruses in breast cancer as a case study, we assess model performance with both zero-shot and in-context learning (ICL; few-shot prompting) strategies against frontier proprietary Large Language Models (LLMs). Llama 3 and Qwen2.5 outperform GPT-5 (API, low/high effort), Gemini 3 Pro Preview, and Meerkat in zero-shot settings, though trailing Gemini 2.5 Pro. ICL leads to improved performance on a case-by-case basis, allowing Llama 3 and Qwen2.5 to match Gemini 2.5 Pro in binary classification. Systematic lexical-ablation experiments show that SLM decisions are often grounded in valid scientific cues but can be influenced by spurious textual artifacts, underscoring need for interpretability in high-stakes pipelines. Our results reveal both promise and limitations of modern SLMs for scientific triage; pairing SLMs with simple but principled prompting strategies can approach performance of the strongest LLMs for targeted literature filtering in co-scientist pipelines.

Figures (7)

• Figure 1: Heatmap performance comparison of zero-shot, zero-shot variants and ICL (few-shot) techniques over the test set examples for 32 model-technique configurations. The top row gives the ground truth label with relevance classifications color-coded as: RELEVANT (green), SOMEWHAT RELEVANT (yellow), IRRELEVANT (red), and NO CLASSIFICATION (gray). The classification labels were aggregated for techniques with multiple runs (Random, Fair, BFS-RS). We see that most models struggled to correctly predict the SOMEWHAT RELEVANT label accurately while retaining good performance on the RELEVANT class. GPT-5 was uniformly conservative, mostly classifying RELEVANT papers as SOMEWHAT RELEVANT. In contrast GPT-5-mini performed better. Note that all models successfully predicted a label — none produced a No Classification result. Models with the ‘-zero’ suffix denote the vanilla zero-shot setting.
• Figure 2: Comparison of sensitivity (recall) vs. precision across all models (shown by symbol colour) and in-context learning modes (symbol shape). Left: Binary classification performance with RELEVANT as the positive class and SOMEWHAT RELEVANT and IRRELEVANT grouped as the negative class. Right: Binary classification performance with RELEVANT and SOMEWHAT RELEVANT grouped as the positive class and IRRELEVANT as the negative class. These settings reflect different priorities that may emerge depending on the number of retrieved papers for a Microbe–Cancer Pair (MCP). A small number of returned papers suggests prioritizing sensitivity and combining RELEVANT and SOMEWHAT RELEVANT papers ( case 2) to pass forward (right panel), while a large number of returned papers may prioritise precision and consider only RELEVANT papers ( case 1) to pass forward (left panel). We observe a clear precision-sensitivity trade-off for LLama 3 (Left panel), with multiple models achieving near-perfect precision. Qwen2.5 Rationale (orange star, under light purple circle) and Llama 3 Fair (blue triangle) closely match Gemini 2.5 Pro in this case. On the other hand (right panel), BFS-RS for Qwen2.5 (orange cross) and Llama 3 Central (blue diamond) perform exceptionally well as they approach the performance of the best commercial frontier models (Gemini 2.5 Pro and GPT-5 Thinking Mini).
• Figure 3: Word removal analysis for Paper 91 with Qwen2.5. The highlighted words represent those removed during analysis, with each color denoting the new class assigned after removal (green for RELEVANT, orange for SOMEWHAT RELEVANT, red for IRRELEVANT, and purple for NO CLASSIFICATION). As expected, removing the key term “MMTV” decreased relevance scores, but removing other relevant terms (e.g.“microRNAs”) instead unexpectedly increased relevance
• Figure 4: Word removal analysis for Paper 26 with Qwen2.5. he highlighted words represent those removed during analysis, with each color denoting the new class assigned after removal (green for RELEVANT, orange for SOMEWHAT RELEVANT, red for IRRELEVANT, and purple for NO CLASSIFICATION). Removing key terms (e.g., “MMTV-like env sequences”) unexpectedly increased relevance scores, and removing irrelevant words (e.g., “countries”) produced the same effect. This highlights the model’s inconsistent and counterintuitive sensitivity to input terms.
• Figure 5: Word removal analysis for Paper 26 with Llama 3. The highlighted words represent those removed during analysis, with each color denoting the new class assigned after removal (green for RELEVANT, orange for SOMEWHAT RELEVANT, red for IRRELEVANT, and purple for NO CLASSIFICATION). Removing the word “No” shifted the paper classification to relevant, suggesting a bias toward positive findings. The same shift occurred when removing both key topic terms (e.g., “MMTV-like env sequences”) and unrelated words (e.g., “specimens”, “evidence”), highlighting inconsistent sensitivity, also observed in Qwen2.5.