Semi-supervised Anomaly Detection via Adaptive Reinforcement Learning-Enabled Method with Causal Inference for Sensor Signals

TL;DR

Tri-CRLAD tackles semi-supervised anomaly detection in sensor signals by integrating counterfactual causal inference with soft actor-critic reinforcement learning. It introduces a Causal Feature Extractor to remove confounding and a triple decision support mechanism—historical-similarity sampling, adaptive threshold smoothing, and adaptive rewards—to enhance exploration and adaptation in changing environments. The approach achieves superior performance over nine baselines across seven datasets, with up to a 23% gain in anomaly-detection stability when labeled anomalies are scarce, and is validated through extensive ablations. This yields a robust, flexible framework with practical implications for reliable sensor monitoring in smart manufacturing, with available code at the provided GitHub repository.

Abstract

Semi-supervised anomaly detection for sensor signals is critical in ensuring system reliability in smart manufacturing. However, existing methods rely heavily on data correlation, neglecting causality and leading to potential misinterpretations due to confounding factors. Moreover, while current reinforcement learning-based methods can effectively identify known and unknown anomalies with limited labeled samples, these methods still face several challenges, such as under-utilization of priori knowledge, lack of model flexibility, and deficient reward feedback during environmental interactions. To address the above problems, this paper innovatively constructs a counterfactual causal reinforcement learning model, termed Triple-Assisted Causal Reinforcement Learning Anomaly Detector (Tri-CRLAD). The model leverages causal inference to extract the intrinsic causal feature in data, enhancing the agent's utilization of prior knowledge and improving its generalization capability. In addition, Tri-CRLAD features a triple decision support mechanism, including a sampling strategy based on historical similarity, an adaptive threshold smoothing adjustment strategy, and an adaptive decision reward mechanism. These mechanisms further enhance the flexibility and generalization ability of the model, enabling it to effectively respond to various complex and dynamically changing environments. Experimental results across seven diverse sensor signal datasets demonstrate that Tri-CRLAD outperforms nine state-of-the-art baseline methods. Notably, Tri-CRLAD achieves up to a 23\% improvement in anomaly detection stability with minimal known anomaly samples, highlighting its potential in semi-supervised anomaly detection scenarios. Our code is available at https://github.com/Aoudsung/Tri-CRLAD.

Figures (8)

• Figure 1: Visualization of the effects of causal inference and sampling strategies. Subfigures \ref{['fig1a']} and \ref{['fig1b']} present the original versus causally inferred data features using t-SNE. Subfigures \ref{['fig1c']} and \ref{['fig1d']} depict the agent's frequency of visits to the same point over identical step counts, employing distance-based and historical similarity-based sampling strategies. The x-axis represents visit frequency, and the y-axis shows the count of points with equal visits. A higher y value at a lower x value suggests a superior sampling strategy. Subfigures \ref{['fig1a']} and \ref{['fig1b']} highlight the efficacy of causally processed features in identifying anomaly samples. Meanwhile, subfigures \ref{['fig1c']} and \ref{['fig1d']} demonstrate the proposed sampling strategy's effectiveness in facilitating a broader exploration of data points within a fixed step count, reducing the redundancy of revisiting identical points.
• Figure 2: The constructed causal graph. Figure(a) and Figure(b) represent the causal graph and the causal graph after using the counterfactual intervention.
• Figure 3: The overall structure of the Tri-CRLAD. To ensure the reliability of the collected sensor signal data, Tri-CRLAD comprises three principal components: the Causal Feature Extraction module, the DRL agent based on the SAC algorithm, and the ADIE module.
• Figure 4: Performance of Different Methods on Multi-class of anomaly Datasets in Scenario 1. Figure(a)(b)(c) shows the performance of Tri-CRLAD in Multi_annthyroid, Multi_cardio, and Multi_har when the anomalies ratio is fixed and the contamination ratio is gradually increased
• Figure 5: Performance of Different Methods on Multi-class of anomaly Datasets in Scenario 2. Figure(a)(b)(c) shows the performance of Tri-CRLAD in Multi_annthyroid, Multi_cardio, and Multi_har data when the contamination ratio is fixed and the anomalies ratio is gradually increased