COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domain
Dimitrios P. Panagoulias, Persephone Papatheodosiou, Anastasios P. Palamidas, Mattheos Sanoudos, Evridiki Tsoureli-Nikita, Maria Virvou, George A. Tsihrintzis
TL;DR
COGNET-MD addresses the evaluation gap for medical-domain LLMs by offering an independent benchmark built around a dataset of 542 MCQs across multiple specialties. The framework combines a multi-domain MCQ dataset with a varying-difficulty scoring scheme, including Partial Credit ($0.5$), Full Credit ($1.0$), and penalties for incorrect answers, and standardizes evaluation through a Benchmark Card and use-case categories (Specialty, Beta, Production). Data and code are hosted on HuggingFace, enabling public access and community contributions. This work advances safe, usable, and reproducible LLM assessment in medicine and provides a foundation for leaderboards and cross-domain comparisons.
Abstract
Large Language Models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence (AI) technology which is rapidly evolving and promises to aid in medical diagnosis either by assisting doctors or by simulating a doctor's workflow in more advanced and complex implementations. In this technical paper, we outline Cognitive Network Evaluation Toolkit for Medical Domains (COGNET-MD), which constitutes a novel benchmark for LLM evaluation in the medical domain. Specifically, we propose a scoring-framework with increased difficulty to assess the ability of LLMs in interpreting medical text. The proposed framework is accompanied with a database of Multiple Choice Quizzes (MCQs). To ensure alignment with current medical trends and enhance safety, usefulness, and applicability, these MCQs have been constructed in collaboration with several associated medical experts in various medical domains and are characterized by varying degrees of difficulty. The current (first) version of the database includes the medical domains of Psychiatry, Dentistry, Pulmonology, Dermatology and Endocrinology, but it will be continuously extended and expanded to include additional medical domains.