Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taming Silent Failures: A Framework for Verifiable AI Reliability

Guan-Yan Yang, Farn Wang

TL;DR

The paper tackles silent AI failures in safety-critical systems by introducing the Formal Assurance and Monitoring Environment (FAME), a framework that combines offline formal specification and monitor synthesis with online runtime enforcement. It uses Signal Temporal Logic (STL) to encode time-bounded safety contracts and synthesizes lightweight monitors via an RTAMT-based toolchain, wrapping opaque AI components with a verifiable safety net. A proof-of-concept in autonomous perception (YOLOv4 in CARLA) demonstrates that FAME detects 93.5% of silent safety violations in challenging scenarios while maintaining nominal overhead and zero false positives, and it provides a concrete path to ISO 26262 and ISO/PAS 8800 certification through structured assurance workflows. The work highlights a pragmatic shift from pre-deployment validation to a continuous, contract-based runtime assurance that can guide retraining, specification refinement, and end-to-end safety across perception–planning–control stacks. Future directions include integrating generative assurance, self-adapting monitors, and compositional assurance to extend FAME across diverse domains and lifecycle stages.

Abstract

The integration of Artificial Intelligence (AI) into safety-critical systems introduces a new reliability paradigm: silent failures, where AI produces confident but incorrect outputs that can be dangerous. This paper introduces the Formal Assurance and Monitoring Environment (FAME), a novel framework that confronts this challenge. FAME synergizes the mathematical rigor of offline formal synthesis with the vigilance of online runtime monitoring to create a verifiable safety net around opaque AI components. We demonstrate its efficacy in an autonomous vehicle perception system, where FAME successfully detected 93.5% of critical safety violations that were otherwise silent. By contextualizing our framework within the ISO 26262 and ISO/PAS 8800 standards, we provide reliability engineers with a practical, certifiable pathway for deploying trustworthy AI. FAME represents a crucial shift from accepting probabilistic performance to enforcing provable safety in next-generation systems.

Taming Silent Failures: A Framework for Verifiable AI Reliability

TL;DR

The paper tackles silent AI failures in safety-critical systems by introducing the Formal Assurance and Monitoring Environment (FAME), a framework that combines offline formal specification and monitor synthesis with online runtime enforcement. It uses Signal Temporal Logic (STL) to encode time-bounded safety contracts and synthesizes lightweight monitors via an RTAMT-based toolchain, wrapping opaque AI components with a verifiable safety net. A proof-of-concept in autonomous perception (YOLOv4 in CARLA) demonstrates that FAME detects 93.5% of silent safety violations in challenging scenarios while maintaining nominal overhead and zero false positives, and it provides a concrete path to ISO 26262 and ISO/PAS 8800 certification through structured assurance workflows. The work highlights a pragmatic shift from pre-deployment validation to a continuous, contract-based runtime assurance that can guide retraining, specification refinement, and end-to-end safety across perception–planning–control stacks. Future directions include integrating generative assurance, self-adapting monitors, and compositional assurance to extend FAME across diverse domains and lifecycle stages.

Abstract

The integration of Artificial Intelligence (AI) into safety-critical systems introduces a new reliability paradigm: silent failures, where AI produces confident but incorrect outputs that can be dangerous. This paper introduces the Formal Assurance and Monitoring Environment (FAME), a novel framework that confronts this challenge. FAME synergizes the mathematical rigor of offline formal synthesis with the vigilance of online runtime monitoring to create a verifiable safety net around opaque AI components. We demonstrate its efficacy in an autonomous vehicle perception system, where FAME successfully detected 93.5% of critical safety violations that were otherwise silent. By contextualizing our framework within the ISO 26262 and ISO/PAS 8800 standards, we provide reliability engineers with a practical, certifiable pathway for deploying trustworthy AI. FAME represents a crucial shift from accepting probabilistic performance to enforcing provable safety in next-generation systems.
Paper Structure (34 sections, 2 equations, 3 figures, 2 tables)

This paper contains 34 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Positioning against the State-of-the-Art
  2. White-Box Formal Verification of DNNs
  3. Robustness, Out-of-Distribution (OOD), and Uncertainty.
  4. Runtime Verification and Assurance
  5. The FAME Framework: A Two-Phase Architecture for Continuous Assurance
  6. Phase 1: Design-Time Specification & Synthesis
  7. Formal Specification: The Language of Safety
  8. Specification Engineering and Proactive Stressing
  9. Controlled NL-to-STL Translation
  10. Automated Monitor Synthesis Toolchain
  11. Phase 2: Run-Time Monitoring & Mitigation
  12. In-Situ Monitoring: The Watchful Eye
  13. Violation Detection & Mitigation Strategy Activation
  14. Macro-Explainability Payload
  15. The Assurance Feedback Loop: Learning from Failure
  16. ...and 19 more sections

Figures (3)

  • Figure 1: The FAME framework integrates offline synthesis of monitors from formal specifications with online enforcement and a feedback loop for continuous assurance.
  • Figure 2: FAME's macro-explainability versus low-level XAI. While low-level XAI seeks to explain why an AI made a specific decision (e.g., pixel importance), FAME's macro-explainability definitively answers which system-level safety rule was violated, providing actionable intelligence for reliability engineers.
  • Figure 3: Comparison of DNN silent failures versus violations detected by the FAME monitor in 100 challenging scenarios. FAME provided a crucial safety warning in over 93% of the failure cases.