Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

Peigen Liu, Rui Ding, Yuren Mao, Ziyan Jiang, Yuxiang Ye, Yunjun Gao, Ying Zhang, Renjie Sun, Longbin Lai, Zhengping Qian

Abstract

Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. However, there is currently no dedicated arena for evolving and benchmarking LLM-based CI. To address this gap, we introduce OpenHospital, an interactive arena where physician agents can evolve CI through interactions with patient agents. This arena employs a data-in-agent-self paradigm that rapidly enhances agent capabilities and provides robust evaluation metrics for benchmarking both medical proficiency and system efficiency. Experiments demonstrate the effectiveness of OpenHospital in both fostering and quantifying CI.

OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

Abstract

Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. However, there is currently no dedicated arena for evolving and benchmarking LLM-based CI. To address this gap, we introduce OpenHospital, an interactive arena where physician agents can evolve CI through interactions with patient agents. This arena employs a data-in-agent-self paradigm that rapidly enhances agent capabilities and provides robust evaluation metrics for benchmarking both medical proficiency and system efficiency. Experiments demonstrate the effectiveness of OpenHospital in both fostering and quantifying CI.
Paper Structure (27 sections, 6 figures, 1 table)

This paper contains 27 sections, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-based Multi-Agent Systems
  4. Benchmarks for LLM-based Multi-Agent Systems
  5. Multi-Agent Systems for Healthcare
  6. Construction of Patient Agents
  7. Clinical Correctness
  8. Persona Diversity
  9. Linguistic Fluency
  10. Behavioral Realism
  11. Benchmarking Agent Evolution
  12. Evaluation Metrics
  13. Medical Capability Metrics
  14. System Efficiency Metric
  15. Experimental Setup
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Interactional response diversity of patient agents across the fixed question set.
  • Figure 2: Dynamic response quality scores evaluated by GPT-5.2 across Accuracy, Relevance, and Persona Alignment.
  • Figure 3: Diagnostic evolution of a physician agent across training batches.
  • Figure 4: Performance trajectories of clinical capabilities across training batches.
  • Figure 5: Trend of Total Token Consumption across training batches.
  • ...and 1 more figures