AgentScore: Autoformulation of Deployable Clinical Scoring Systems

TL;DR

AgentScore addresses the gap between high-performing clinical models and real-world deployability by learning unit-weighted, interpretable checklists under strict guideline-compatible constraints. It uses an LLM to propose clinically meaningful rules, coupled with a deterministic verification loop that enforces performance, sparsity, and rule diversity, producing a compact checklist $S(\tilde{\mathbf{x}}) = \sum_{r_j \in \mathcal{S}} r_j(\tilde{\mathbf{x}})$ with a threshold $\tau$. Across eight real-world tasks from MIMIC-IV and eICU, AgentScore matches or exceeds state-of-the-art score-learning baselines under deployability constraints and outperforms guideline-based scores on external validation, while remaining suitable for bedside use. Clinician evaluations further favor AgentScore checklists for guideline alignment, ease of bedside application, and deployment readiness, underscoring the practical impact of constrained semantic optimization for guideline-conformant risk scoring. The work suggests a generalizable path for integrating semantic rule generation with rigorous data-grounded validation to produce auditable, deployable decision aids in safety-critical settings.

Abstract

Modern clinical practice relies on evidence-based guidelines implemented as compact scoring systems composed of a small number of interpretable decision rules. While machine-learning models achieve strong performance, many fail to translate into routine clinical use due to misalignment with workflow constraints such as memorability, auditability, and bedside execution. We argue that this gap arises not from insufficient predictive power, but from optimizing over model classes that are incompatible with guideline deployment. Deployable guidelines often take the form of unit-weighted clinical checklists, formed by thresholding the sum of binary rules, but learning such scores requires searching an exponentially large discrete space of possible rule sets. We introduce AgentScore, which performs semantically guided optimization in this space by using LLMs to propose candidate rules and a deterministic, data-grounded verification-and-selection loop to enforce statistical validity and deployability constraints. Across eight clinical prediction tasks, AgentScore outperforms existing score-generation methods and achieves AUC comparable to more flexible interpretable models despite operating under stronger structural constraints. On two additional externally validated tasks, AgentScore achieves higher discrimination than established guideline-based scores.

Figures (8)

• Figure 1: Clinical scoring systems: Guideline artifacts are compact, explicit checklists intended for reliable manual use.
• Figure 2: Overview of AgentScore. An LLM-based proposal agent generates candidate rules from a dataset description and tool-mediated aggregate statistics; it never receives patient-level records. Proposed rules are screened by a deterministic validation module enforcing statistical performance and grammar-level deployability constraints (e.g., complexity limits, unit weights). Statistically admissible rules are reviewed by a clinical plausibility agent to filter semantically incoherent proposals, after which a score assembly agent selects a compact clinical checklist from the retained pool and the evaluation tools choose the operating threshold.
• Figure 3: External validation against guidelines.AgentScore outperforms clinical guidelines in AUROC (mean $\pm$ std over 5 seeds; guidelines deterministic).
• Figure 4: Effect of LLM backbone choice. Performance of AgentScore across different language-model backbones. While GPT-5 attains the strongest results on average, GPT-4o and DeepSeek V3.2 exhibit comparable performance trends, suggesting limited sensitivity to the specific LLM used for rule proposal.
• Figure 5: Risk monotonicity across score values. Empirical outcome prevalence (percentage positive) as a function of the discrete AgentScore value.

Theorems & Definitions (1)

• Definition 2.1: Deployable Clinical Scoring System