Grounded Multilingual Medical Reasoning for Question Answering with Large Language Models

TL;DR

The paper tackles multilingual, reasoning-grounded medical QA by creating a pipeline that builds parallel medical knowledge bases from Wikipedia, retrieves relevant evidence, and generates ground-truth reasoning traces conditioned on the correct answer. It produces a large multilingual traces dataset and demonstrates consistent improvements in in-context learning and supervised fine-tuning across English, Italian, and Spanish, achieving state-of-the-art results for 8B-parameter LLMs. The approach also emphasizes grounding in factual medical knowledge and provides extensive evaluation, including out-of-domain testing and error analysis. The work releases the reasoning traces, translated QA datasets, and knowledge resources to support safer, more transparent multilingual clinical decision-support tools. Overall, multilingual trace-grounding and retrieval augmentation emerge as effective strategies to enhance medical QA across languages.

Abstract

Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces grounded in factual medical knowledge. We produce 500k traces in English, Italian, and Spanish, using a retrievalaugmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and outof-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of safer, more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.

Figures (8)

• Figure 1: Schema of our proposed pipeline to generate reasoning traces for multilingual multiple-choice medical question answering (QA). First, we create a Knowledge Base (KB) of medical information for each language. We extract relevant chunks from the KB for each pair of Question-Options (QO pair) in the source QA datasets, which we automatically port from English into Italian and Spanish. We prompt an LLM with the retrieved chunks and the QO pair for context rearrangement. Finally, we utilize the rearranged context, the question, the options and the correct answer to generate a reasoning trace that answers the question itself. Answers that lead to the wrong conclusion are dropped, while the remaining form our reasoning traces dataset.
• Figure 2: Example of a reasoning trace given a question and the options. The generated trace is composed of 3 parts. First (violet), there is a search for potential paths and useful knowledge required to answer the question, which are then utilised to elaborate on the provided options (green part). Finally (red), the final answer is reported as conclusion.
• Figure 3: Impact of our traces via in-context learning per dataset and language. Each boxplot represents the distribution of the improvement due to exposing models to our traces as two-shot examples for the eleven tested open models. The improvement is measured in terms of an increase in accuracy on the testing splits of the datasets. The baseline is represented by the same model prompted via two-shot, where each example contains the correct answer without our generated reasoning.
• Figure 4: Comparison of our traces with previous work and impact of multilingual fine-tuning. The top-left box-plots represent the distribution of the average improvement in accuracy among $11$ open-source models by trace type when provided at inference time (two-shot). The top-right histogram shows the average impact of different types of traces when provided at fine-tuning time. The bottom-left histogram shows the performances of the best models proposed in the literature against our fine-tuned version of Qwen3-8B ("OUR FT") and the same model further enhanced with $2$ of our traces. The bottom-right histogram reports on the impact of fine-tuning on single-language ('Only ES/EN/IT") versus training on all the languages ("ES+EN+IT").
• Figure 5: Example of the expert analysis on a model answer which presents a critical clinical knowledge mistake.