Context-Awareness and Interpretability of Rare Occurrences for Discovery and Formalization of Critical Failure Modes

TL;DR

This paper presents CAIRO, an ontology-based, human-in-the-loop framework to discover and formalize Critical Phenomena (CP) in autonomous-vehicle perception. It combines multi-source detection, feature extraction, and ontology graphs (OWL/XML) with DL/SWRL reasoning to enable queryable, explainable V&V for rare failure modes, including adversarial CP. The approach formalizes CP via T-box/A-box axioms and SWRL rules within a hybrid OWA/CWA setting, enabling systematic root-cause analysis and knowledge sharing across domains. Experiments on CityScape and Driving Downtown demonstrate CP identification under occlusion, misdetections, and adversarial/noise perturbations, revealing insights into model weaknesses and how HITL can guide robust improvements. Overall, CAIRO advances interpretability, accountability, and scalable collaboration for safer, more reliable CV-enabled critical systems.

Abstract

Vision systems are increasingly deployed in critical domains such as surveillance, law enforcement, and transportation. However, their vulnerabilities to rare or unforeseen scenarios pose significant safety risks. To address these challenges, we introduce Context-Awareness and Interpretability of Rare Occurrences (CAIRO), an ontology-based human-assistive discovery framework for failure cases (or CP - Critical Phenomena) detection and formalization. CAIRO by design incentivizes human-in-the-loop for testing and evaluation of criticality that arises from misdetections, adversarial attacks, and hallucinations in AI black-box models. Our robust analysis of object detection model(s) failures in automated driving systems (ADS) showcases scalable and interpretable ways of formalizing the observed gaps between camera perception and real-world contexts, resulting in test cases stored as explicit knowledge graphs (in OWL/XML format) amenable for sharing, downstream analysis, logical reasoning, and accountability.

Figures (8)

• Figure 1: Testing & evaluation (T&E) with Knowledge Graphs (KGs) in green and human-in-the-loop (HITL) interpreting detection model inferences in blue through CAIRO processes in red.
• Figure 2: OWA, CWA, and hybrid approach comparison in KG representation.
• Figure 3: CAIRO pipeline: A comprehensive framework integrating KGs, logical reasoning, and HITL validation to enhance the reliability and interpretability of critical phenomena detection in autonomous systems.
• Figure 4: Demonstration of a set of behavioral CP: camera mis-perception and human CP. CP_0003 (green) shows behavioral CP - illegitimate use of a crosswalk by the bicyclists. CP_0005 (purple) finds behavioral CP - detached wheels being close to vehicle lane. CP_0001(orange) finds a dynamic CP - vulnerable road users that are highly occluded.
• Figure 5: CP_0002 from Table \ref{['tab:cp_queries_combined']} showing temporal relations, highlighting objects (e.g., stroller) that appears in one time frame but missing in the consequent one.