Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causal Characterization of Measurement and Mechanistic Anomalies

Hendrik Suhr, David Kaltenpoth, Jilles Vreeken

TL;DR

This work tackles explainable anomaly detection by distinguishing measurement errors from genuine mechanistic shifts within a causal framework over latent and observed variables $\mathbf{X}^*$ and $\mathbf{X}$. It introduces a latent interventional model with hard interventions on latent nodes and on observed measurements, proves structural identifiability in the infinite-sample limit, and develops Cali, a latent maximum-likelihood estimator that marginalizes over unobserved clean values via Monte Carlo and leverages robust ANM-based causal mechanisms under Sparse Mechanism Shifts. Cali localizes root causes and classifies anomaly types, achieving state-of-the-art performance in synthetic and real datasets (Sachs, Causal Chambers) and providing interpretable case studies such as a NYC Taxi dataset. The method remains robust to unknown DAGs by combining causal discovery with the latent-MLE framework, offering a practical tool for RCA in noisy data and enabling actionable distinctions between measurement corrections and genuine process changes.

Abstract

Root cause analysis of anomalies aims to identify those features that cause the deviation from the normal process. Existing methods ignore, however, that anomalies can arise through two fundamentally different processes: measurement errors, where data was generated normally but one or more values were recorded incorrectly, and mechanism shifts, where the causal process generating the data changed. While measurement errors can often be safely corrected, mechanistic anomalies require careful consideration. We define a causal model that explicitly captures both types by treating outliers as latent interventions on latent ("true") and observed ("measured") variables. We show that they are identifiable, and propose a maximum likelihood estimation approach to put this to practice. Experiments show that our method matches state-of-the-art performance in root cause localization, while it additionally enables accurate classification of anomaly types, and remains robust even when the causal DAG is unknown.

Causal Characterization of Measurement and Mechanistic Anomalies

TL;DR

This work tackles explainable anomaly detection by distinguishing measurement errors from genuine mechanistic shifts within a causal framework over latent and observed variables and . It introduces a latent interventional model with hard interventions on latent nodes and on observed measurements, proves structural identifiability in the infinite-sample limit, and develops Cali, a latent maximum-likelihood estimator that marginalizes over unobserved clean values via Monte Carlo and leverages robust ANM-based causal mechanisms under Sparse Mechanism Shifts. Cali localizes root causes and classifies anomaly types, achieving state-of-the-art performance in synthetic and real datasets (Sachs, Causal Chambers) and providing interpretable case studies such as a NYC Taxi dataset. The method remains robust to unknown DAGs by combining causal discovery with the latent-MLE framework, offering a practical tool for RCA in noisy data and enabling actionable distinctions between measurement corrections and genuine process changes.

Abstract

Root cause analysis of anomalies aims to identify those features that cause the deviation from the normal process. Existing methods ignore, however, that anomalies can arise through two fundamentally different processes: measurement errors, where data was generated normally but one or more values were recorded incorrectly, and mechanism shifts, where the causal process generating the data changed. While measurement errors can often be safely corrected, mechanistic anomalies require careful consideration. We define a causal model that explicitly captures both types by treating outliers as latent interventions on latent ("true") and observed ("measured") variables. We show that they are identifiable, and propose a maximum likelihood estimation approach to put this to practice. Experiments show that our method matches state-of-the-art performance in root cause localization, while it additionally enables accurate classification of anomaly types, and remains robust even when the causal DAG is unknown.
Paper Structure (62 sections, 2 theorems, 23 equations, 21 figures, 5 tables, 2 algorithms)

This paper contains 62 sections, 2 theorems, 23 equations, 21 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Causal Model
  4. Causal Anomalousness
  5. Theory
  6. Structural Identifiability
  7. Latent Maximum Likelihood Estimation
  8. Marginalization
  9. MLE objective
  10. Instantiation
  11. Related Work
  12. Experiments
  13. Data
  14. Instantiation of Cali
  15. Evaluation Metrics
  16. ...and 47 more sections

Key Result

Theorem 3.4

Let $\mathbf A$ be the true assignment, and let ${\mathbf A'}$ be an arbitrary latent assignment. Suppose that, for almost every assignment value $\mathbf a$, the conditional distribution $P_{\mathbf{X} \mid \mathbf A=\mathbf a}$ is faithful to ${\mathcal{G}}(\mathbf a)$. Then we have that ${\mathbf

Figures (21)

  • Figure 1: Different outlier types under the causal DAG $X \to Y$, $X \to W$, and $Y \to Z$. While the mechanistic outlier $\mathbf g$ only stands out when considering the $X \to Y$ relationship, the measurement outlier $\textcolor{OIblue}{$\mathbf m$}\xspace$ stands out for $X \to Y$ and $Y \to Z$.
  • Figure 2: Different outlier types induce different downstream behavior. Red lightning symbols indicate interventions that correspond to measurement outliers (on observed variables) and mechanistic outliers (on latent variables).
  • Figure 3: Root Cause Localization (Top-$k$ recall, higher is better) for synthetic data (left) and real data (right). Outlier strength corresponds to mean shifts in terms of standard deviations (left), or the relative strength of intervention (right): low (Sachs), medium (Causal Chamber, mid), high (Causal Chamber, strong). We set $k$ to the true number of root causes in each sample. Shaded regions indicate 95% confidence intervals across 20 runs.
  • Figure 4: Classification accuracy (higher is better) of outlier types for synthetic data (left) and real world data (right). Outlier strength corresponds to mean shifts in terms of standard deviations (left), or the relative strength of intervention (right): low (Sachs), medium (Causal Chamber, mid), high (Causal Chamber, strong). Random guessing corresponds to 50%. Shaded regions indicate 95% confidence intervals across 20 runs.
  • Figure 5: Causal relationships of the NYC Taxi Dataset.
  • ...and 16 more figures

Theorems & Definitions (9)

  • Definition 2.1: Mechanistic Anomalies
  • Definition 2.2: Measurement Anomalies
  • Definition 3.1: Equivalence of DAGs
  • Definition 3.2: Structural equivalence of assignments
  • Definition 3.3: Respect of CI constraints
  • Theorem 3.4: Structural Identifiability
  • Theorem : Structural Identifiability
  • proof : Proof of Theorem \ref{['thm:asymptotic-recovery']}
  • Example 1: Observational equivalence in the bivariate setting