Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IMB: An Italian Medical Benchmark for Question Answering

Antonio Romano, Giuseppe Riccio, Mariano Barone, Marco Postiglione, Vincenzo Moscato

TL;DR

IMB introduces two Italian biomedical QA benchmarks, IMB-QA and IMB-MCQA, to address the scarcity of non-English resources for medical QA and to capture both informal forum dialogue and formal exam content. The authors demonstrate that retrieval-augmented generation and in-domain fine-tuning can outperform mere model scaling, with specialized Italian models and domain-adaptive strategies delivering strong performance on MCQA and open-ended QA tasks. Through rigorous preprocessing, anonymization, and topic modeling, IMB provides a large, diverse, and realistic testbed for multilingual medical QA and retrieval-based methods. The work highlights the practical importance of domain knowledge and efficient information retrieval for reliable medical AI, and it releases the datasets and evaluation pipelines to foster further research in multilingual clinical QA.

Abstract

Online medical forums have long served as vital platforms where patients seek professional healthcare advice, generating vast amounts of valuable knowledge. However, the informal nature and linguistic complexity of forum interactions pose significant challenges for automated question answering systems, especially when dealing with non-English languages. We present two comprehensive Italian medical benchmarks: \textbf{IMB-QA}, containing 782,644 patient-doctor conversations from 77 medical categories, and \textbf{IMB-MCQA}, comprising 25,862 multiple-choice questions from medical specialty examinations. We demonstrate how Large Language Models (LLMs) can be leveraged to improve the clarity and consistency of medical forum data while retaining their original meaning and conversational style, and compare a variety of LLM architectures on both open and multiple-choice question answering tasks. Our experiments with Retrieval Augmented Generation (RAG) and domain-specific fine-tuning reveal that specialized adaptation strategies can outperform larger, general-purpose models in medical question answering tasks. These findings suggest that effective medical AI systems may benefit more from domain expertise and efficient information retrieval than from increased model scale. We release both datasets and evaluation frameworks in our GitHub repository to support further research on multilingual medical question answering: https://github.com/PRAISELab-PicusLab/IMB.

IMB: An Italian Medical Benchmark for Question Answering

TL;DR

IMB introduces two Italian biomedical QA benchmarks, IMB-QA and IMB-MCQA, to address the scarcity of non-English resources for medical QA and to capture both informal forum dialogue and formal exam content. The authors demonstrate that retrieval-augmented generation and in-domain fine-tuning can outperform mere model scaling, with specialized Italian models and domain-adaptive strategies delivering strong performance on MCQA and open-ended QA tasks. Through rigorous preprocessing, anonymization, and topic modeling, IMB provides a large, diverse, and realistic testbed for multilingual medical QA and retrieval-based methods. The work highlights the practical importance of domain knowledge and efficient information retrieval for reliable medical AI, and it releases the datasets and evaluation pipelines to foster further research in multilingual clinical QA.

Abstract

Online medical forums have long served as vital platforms where patients seek professional healthcare advice, generating vast amounts of valuable knowledge. However, the informal nature and linguistic complexity of forum interactions pose significant challenges for automated question answering systems, especially when dealing with non-English languages. We present two comprehensive Italian medical benchmarks: \textbf{IMB-QA}, containing 782,644 patient-doctor conversations from 77 medical categories, and \textbf{IMB-MCQA}, comprising 25,862 multiple-choice questions from medical specialty examinations. We demonstrate how Large Language Models (LLMs) can be leveraged to improve the clarity and consistency of medical forum data while retaining their original meaning and conversational style, and compare a variety of LLM architectures on both open and multiple-choice question answering tasks. Our experiments with Retrieval Augmented Generation (RAG) and domain-specific fine-tuning reveal that specialized adaptation strategies can outperform larger, general-purpose models in medical question answering tasks. These findings suggest that effective medical AI systems may benefit more from domain expertise and efficient information retrieval than from increased model scale. We release both datasets and evaluation frameworks in our GitHub repository to support further research on multilingual medical question answering: https://github.com/PRAISELab-PicusLab/IMB.
Paper Structure (29 sections, 1 equation, 7 figures, 8 tables)

This paper contains 29 sections, 1 equation, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Open-Domain and MRC Biomedical QA
  4. Multiple Choice QA
  5. IMB Dataset
  6. Data Collection
  7. Data preprocessing methods
  8. Preprocessing for IMB-QA
  9. Data cleaning
  10. Text Normalization, Answer Reformulation, and Data Anonymization
  11. Data Categorization
  12. Preprocessing for IMB-MCQA
  13. Data Analysis
  14. Diversity of Questions
  15. Need for Domain-Specific Expertise
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Workflow for the construction of the Italian Medical Benchmark (IMB), consisting of open-ended question-answer pairs (IMB-QA) and multiple-choice question-answer assessments (IMB-MCQA).
  • Figure 2: Percentage of questions with above-average difficulty by macro-category in IMB-QA. The score refers to the percentage of questions in each category that were classified as above-average in difficulty, based on our difficulty index
  • Figure 3: Percentage of questions with above-average difficulty by category in IMB-MCQA. The score refers to the percentage of questions in each category that were classified as above-average in difficulty, based on our difficulty index
  • Figure 4: Distribution of macro-categories in IMB-QA.
  • Figure 5: Distribution of categories in IMB-MCQA.
  • ...and 2 more figures