LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration

TL;DR

LogiDebrief is introduced, an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation.

Abstract

Emergency response services are critical to public safety, with 9-1-1 call-takers playing a key role in ensuring timely and effective emergency operations. To ensure call-taking performance consistency, quality assurance is implemented to evaluate and refine call-takers' skillsets. However, traditional human-led evaluations struggle with high call volumes, leading to low coverage and delayed assessments. We introduce LogiDebrief, an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation. LogiDebrief formalizes call-taking requirements as logical specifications, enabling systematic assessment of 9-1-1 calls against procedural guidelines. It employs a three-step verification process: (1) contextual understanding to identify responder types, incident classifications, and critical conditions; (2) STL-based runtime checking with LLM integration to ensure compliance; and (3) automated aggregation of results into quality assurance reports. Beyond its technical contributions, LogiDebrief has demonstrated real-world impact. Successfully deployed at Metro Nashville Department of Emergency Communications, it has assisted in debriefing 1,701 real-world calls, saving 311.85 hours of active engagement. Empirical evaluation with real-world data confirms its accuracy, while a case study and extensive user study highlight its effectiveness in enhancing call-taking performance.

Figures (3)

• Figure 1: Message Length vs. One-shot Solve Rate with Llama 3.2. This figure shows the relationship between prompt length (in tokens) and solve rate, segmented by primary call categories (Medical, Fire, and Police). Shaded regions indicate standard deviations.
• Figure 2: Illustration of the Quality Assurance Form. Each form $\Psi$ consists of multiple checks ($\varphi$) derived from call-taking protocols. Requirements ($r$) are validated based on preconditions ($\mathcal{P}$) using extracted conversational signals ($\omega$). Aggregated results classify compliance such as Yes, No, (Caller) Refused, and NA.
• Figure 3: Overview of the LogiDebrief Workflow. It evaluates call-taker performance by analyzing past calls against formalized requirements. It extracts conversational signals $\omega_{(a)}$ and $\omega_{(b)}$ from past calls, then applies quality assurance evaluation (QAE) checks. During runtime monitoring with LLMs, it finalizes applicable checks based on call context, checks the compliance of each check, and aggregates results into a quality assurance form with actionable template-based feedback.