Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond

Shubham Kulkarni, Alexander Lyzhov, Shiva Chaitanya, Preetam Joshi

TL;DR

The work addresses the gap between turn-level evaluation and regulatory clinical compliance in AI-assisted healthcare dialogues by introducing Obligatory-Information Phase Structured Compliance Evaluation (OIP--SCE). OIP--SCE models compliance as a structured, sequential engagement over obligatory information phases, using two predicates—Coverage and OrderSafe—and a final CallSuccess decision to ensure that every required phase is completed in the correct order with auditable evidence. The authors provide a formal specification (one-row-per-phase), a clinician–engineer workflow for governance and updates, and two case studies (respiratory history and insurance benefit verification) to demonstrate actionable, auditable results at scale. They also discuss practical deployment considerations, including policy-as-code, human-in-the-loop review, and integration into healthcare governance, with the aim of making AI in healthcare safer, more trustworthy, and interoperable with real clinical workflows.

Abstract

Conversational AI is starting to support real clinical work, but most evaluation methods miss how compliance depends on the full course of a conversation. We introduce Obligatory-Information Phase Structured Compliance Evaluation (OIP-SCE), an evaluation method that checks whether every required clinical obligation is met, in the right order, with clear evidence for clinicians to review. This makes complex rules practical and auditable, helping close the gap between technical progress and what healthcare actually needs. We demonstrate the method in two case studies (respiratory history, benefits verification) and show how phase-level evidence turns policy into shared, actionable steps. By giving clinicians control over what to check and engineers a clear specification to implement, OIP-SCE provides a single, auditable evaluation surface that aligns AI capability with clinical workflow and supports routine, safe use.

All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond

TL;DR

The work addresses the gap between turn-level evaluation and regulatory clinical compliance in AI-assisted healthcare dialogues by introducing Obligatory-Information Phase Structured Compliance Evaluation (OIP--SCE). OIP--SCE models compliance as a structured, sequential engagement over obligatory information phases, using two predicates—Coverage and OrderSafe—and a final CallSuccess decision to ensure that every required phase is completed in the correct order with auditable evidence. The authors provide a formal specification (one-row-per-phase), a clinician–engineer workflow for governance and updates, and two case studies (respiratory history and insurance benefit verification) to demonstrate actionable, auditable results at scale. They also discuss practical deployment considerations, including policy-as-code, human-in-the-loop review, and integration into healthcare governance, with the aim of making AI in healthcare safer, more trustworthy, and interoperable with real clinical workflows.

Abstract

Conversational AI is starting to support real clinical work, but most evaluation methods miss how compliance depends on the full course of a conversation. We introduce Obligatory-Information Phase Structured Compliance Evaluation (OIP-SCE), an evaluation method that checks whether every required clinical obligation is met, in the right order, with clear evidence for clinicians to review. This makes complex rules practical and auditable, helping close the gap between technical progress and what healthcare actually needs. We demonstrate the method in two case studies (respiratory history, benefits verification) and show how phase-level evidence turns policy into shared, actionable steps. By giving clinicians control over what to check and engineers a clear specification to implement, OIP-SCE provides a single, auditable evaluation surface that aligns AI capability with clinical workflow and supports routine, safe use.
Paper Structure (43 sections, 6 equations, 1 table)

This paper contains 43 sections, 6 equations, 1 table.

Table of Contents

  1. Introduction
  2. Contributions and scope.
  3. From Turn-Level Metrics to Compliance-Centered Evaluation
  4. Why turn-level evaluation fails clinical compliance
  5. What clinical compliance actually requires
  6. Compliance as structured, sequential logic:
  7. Operational and regulatory constraints
  8. Principles for a compliance-centered evaluator
  9. Evaluator design, governance, and generalization
  10. From healthcare to general AI evaluation:
  11. Obligatory-Information Phase Structured Compliance Evaluation (OIP--SCE)
  12. Phases and dependency structure
  13. Row schema (one row per phase)
  14. Conventions:
  15. Notation:
  16. ...and 28 more sections