Planner-Auditor Twin: Agentic Discharge Planning with FHIR-Based LLM Planning, Guideline Recall, Optional Caching and Self-Improvement

TL;DR

A self-improving, cache-optional Planner-Auditor framework that improves safety and reliability by decoupling generation from deterministic validation and targeted replay for high-confidence, low-coverage cases is introduced.

Abstract

Objective: Large language models (LLMs) show promise for clinical discharge planning, but their use is constrained by hallucination, omissions, and miscalibrated confidence. We introduce a self-improving, cache-optional Planner-Auditor framework that improves safety and reliability by decoupling generation from deterministic validation and targeted replay. Materials and Methods: We implemented an agentic, retrospective, FHIR-native evaluation pipeline using MIMIC-IV-on-FHIR. For each patient, the Planner (LLM) generates a structured discharge action plan with an explicit confidence estimate. The Auditor is a deterministic module that evaluates multi-task coverage, tracks calibration (Brier score, ECE proxies), and monitors action-distribution drift. The framework supports two-tier self-improvement: (i) within-episode regeneration when enabled, and (ii) cross-episode discrepancy buffering with replay for high-confidence, low-coverage cases. Results: While context caching improved performance over baseline, the self-improvement loop was the primary driver of gains, increasing task coverage from 32% to 86%. Calibration improved substantially, with reduced Brier/ECE and fewer high-confidence misses. Discrepancy buffering further corrected persistent high-confidence omissions during replay. Discussion: Feedback-driven regeneration and targeted replay act as effective control mechanisms to reduce omissions and improve confidence reliability in structured clinical planning. Separating an LLM Planner from a rule-based, observational Auditor enables systematic reliability measurement and safer iteration without model retraining. Conclusion: The Planner-Auditor framework offers a practical pathway toward safer automated discharge planning using interoperable FHIR data access and deterministic auditing, supported by reproducible ablations and reliability-focused evaluation.

Figures (5)

• Figure 1: Overview of Planner-Auditor architecture. The workflow begins with FHIR data retrieval and summarization via the simulation harness. The Planner Agent (GPT-4o-mini) generates a draft plan. A coverage check gate evaluates the draft; high‐confidence plans with missing coverage are flagged into a memory buffer. The downstream Auditing Agent scores coverage, calibration, and drift and records a verdict without blocking real-time generation. In the cross-episode replay step, buffered failures are re-injected for offline refinement to repair the flagged cases.
• Figure 2: (a) Per-episode completion rates for four discharge planning tasks (follow-up, medication, education, monitoring) across system configurations. Follow-up is near ceiling (96-100%). Baseline fails most often on education/monitoring (both 54%). Caching moderately improves them, while self-improvement delivers the largest gains on them (education 54% to 94%, monitoring 54% to 88%). Cache + SI further boosts education (98%) and monitoring (92%). (b) Full discharge coverage (all 4 tasks present simultaneously) shows a staircase: Baseline 32% to Cache 52% to SI / Cache+SI 86% to Buffer Replay 100%. Caching provides a meaningful quality gain and improves latency. Cache + SI matches SI's coverage (86%) more efficiently. Buffer Replay achieves 100% coverage (cautious interpretation due to small N=7 and high latency cost).
• Figure 3: (a) High-confidence coverage failures by configuration. High-confidence error rate (fraction of episodes with coverage failure assigned high confidence) drops with completeness mechanisms: baseline (66%) $\rightarrow$ context caching (46%) $\rightarrow$ self-improvement (14%) and cache+SI (14%), with buffer replay being 0%. (b) Calibration quality. Left, Brier score; right, ECE. Calibration improves sharply with completeness mechanisms: baseline is worst (Brier 0.544, ECE 0.564), improving with context caching (Brier 0.382, ECE 0.356) and substantially with self-improvement (Brier 0.126, ECE 0.062). Cache + self-improve achieves the best ECE (0.034). Buffer replay has the lowest Brier (0.017) but worse ECE (0.107).
• Figure 4: (a) Quality--latency trade-off. Context cache dominates baseline. Self-improvement significantly boosts quality (86% coverage) with latency overhead (19.65s). Cache + SI improves SI: same coverage (86%) at lower latency (18.70s). Buffer replay achieves perfect coverage (100%) but with highest latency (27.78s). (b) Calibration--latency trade-off. ECE decreases dramatically with SI (0.564 $\rightarrow$ 0.062) and Cache+SI (0.034). Buffer replay's ECE (0.107) is worse than SI variants with even higher latency. Cache+SI is the best-calibrated high-coverage operating point. (c) Pareto frontier: accuracy vs latency. Line in dark orange marks the Pareto-efficient frontier, representing configurations minimizing latency while maximizing full coverage.
• Figure 5: Front-end evaluation dashboard for discharge planning system built by React and fastMCP. Interactive web interface shows comparative metrics (coverage, latency, calibration), per-sample results, violations, and real-time execution logs across multiple system configurations (baseline, context cache, self-improvement, and combinations). The dashboard connects to a FastAPI evaluation backend.