Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Doctorina MedBench: End-to-End Evaluation of Agent-Based Medical AI

Anna Kozlova, Stanislau Salavei, Pavel Satalkin, Hanna Plotnitskaya, Sergey Parfenyuk

Abstract

We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions. Unlike traditional medical benchmarks that rely on solving standardized test questions, the proposed approach models a multi-step clinical dialogue in which either a physician or an AI system must collect medical history, analyze attached materials (including laboratory reports, images, and medical documents), formulate differential diagnoses, and provide personalized recommendations. System performance is evaluated using the D.O.T.S. metric, which consists of four components: Diagnosis, Observations/Investigations, Treatment, and Step Count, enabling assessment of both clinical correctness and dialogue efficiency. The system also incorporates a multi-level testing and quality monitoring architecture designed to detect model degradation during both development and deployment. The framework supports safety-oriented trap cases, category-based random sampling of clinical scenarios, and full regression testing. The dataset currently contains more than 1,000 clinical cases covering over 750 diagnoses. The universality of the evaluation metrics allows the framework to be used not only to assess medical AI systems, but also to evaluate physicians and support the development of clinical reasoning skills. Our results suggest that simulation of clinical dialogue may provide a more realistic assessment of clinical competence compared to traditional examination-style benchmarks.

Doctorina MedBench: End-to-End Evaluation of Agent-Based Medical AI

Abstract

We present Doctorina MedBench, a comprehensive evaluation framework for agent-based medical AI based on the simulation of realistic physician-patient interactions. Unlike traditional medical benchmarks that rely on solving standardized test questions, the proposed approach models a multi-step clinical dialogue in which either a physician or an AI system must collect medical history, analyze attached materials (including laboratory reports, images, and medical documents), formulate differential diagnoses, and provide personalized recommendations. System performance is evaluated using the D.O.T.S. metric, which consists of four components: Diagnosis, Observations/Investigations, Treatment, and Step Count, enabling assessment of both clinical correctness and dialogue efficiency. The system also incorporates a multi-level testing and quality monitoring architecture designed to detect model degradation during both development and deployment. The framework supports safety-oriented trap cases, category-based random sampling of clinical scenarios, and full regression testing. The dataset currently contains more than 1,000 clinical cases covering over 750 diagnoses. The universality of the evaluation metrics allows the framework to be used not only to assess medical AI systems, but also to evaluate physicians and support the development of clinical reasoning skills. Our results suggest that simulation of clinical dialogue may provide a more realistic assessment of clinical competence compared to traditional examination-style benchmarks.

Paper Structure

This paper contains 25 sections, 18 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. System Overview
  3. General Principles of the AI Doctor
  4. Simulation of the AI-Patient
  5. Evaluation Framework
  6. Core Metrics (D.O.T.S.)
  7. Evaluator Implementation
  8. Multi-Level Testing Protocol and Continuous Monitoring
  9. Real-Time Quality Monitoring
  10. Level 1 – Trap-Based Testing
  11. Level 2 – Category-Based Random Sampling
  12. Level 3 – Full Regression Testing
  13. Case Design and Dataset Management
  14. Source and Maintenance of Cases
  15. Treatment Evaluation Logic. Categorization of tests.
  16. ...and 10 more sections

Figures (14)

  • Figure 1: AI Doctor evaluation workflow
  • Figure 2: D.O.T.S.
  • Figure 3: Real-time Monitoring System Architecture. Short clinical "trap" test scenarios are executed in parallel with the production system and evaluated using DOTS metrics. Upon detection of anomalies, a comprehensive regression run is triggered. If performance degradation is confirmed, the system automatically dispatches notifications, blocks model promotion, and initiates remediation processes before returning the model to the monitoring cycle.
  • Figure 4: Distribution of cases by clinical category
  • Figure 5: Distribution of cases by age group.
  • ...and 9 more figures