RPG-AE: Neuro-Symbolic Graph Autoencoders with Rare Pattern Mining for Provenance-Based Anomaly Detection

TL;DR

RPG-AE tackles provenance-based anomaly detection for Advanced Persistent Threats by fusing a Graph Autoencoder with rare-pattern mining. It constructs a process-level k-NN graph to learn normal relational structure and uses an Apriori-based module to identify rare co-occurrence patterns, boosting GAE scores with a degree-based rarity signal. Empirical results on DARPA Transparent Computing data show consistent improvements in anomaly ranking over baselines, achieving competitive performance with ensemble methods while offering interpretability through mined rare patterns. The framework demonstrates the value of combining graph representation learning with symbolic pattern mining to enhance effectiveness and interpretability in provenance-based security analytics.

Abstract

Advanced Persistent Threats (APTs) are sophisticated, long-term cyberattacks that are difficult to detect because they operate stealthily and often blend into normal system behavior. This paper presents a neuro-symbolic anomaly detection framework that combines a Graph Autoencoder (GAE) with rare pattern mining to identify APT-like activities in system-level provenance data. Our approach first constructs a process behavioral graph using k-Nearest Neighbors based on feature similarity, then learns normal relational structure using a Graph Autoencoder. Anomaly candidates are identified through deviations between observed and reconstructed graph structure. To further improve detection, we integrate an rare pattern mining module that discovers infrequent behavioral co-occurrences and uses them to boost anomaly scores for processes exhibiting rare signatures. We evaluate the proposed method on the DARPA Transparent Computing datasets and show that rare-pattern boosting yields substantial gains in anomaly ranking quality over the baseline GAE. Compared with existing unsupervised approaches on the same benchmark, our single unified model consistently outperforms individual context-based detectors and achieves performance competitive with ensemble aggregation methods that require multiple separate detectors. These results highlight the value of coupling graph-based representation learning with classical pattern mining to improve both effectiveness and interpretability in provenance-based security anomaly detection.

Figures (3)

• Figure 1: Overview of the proposed RPG-AE architecture. The pipeline first extracts process-level behavioral features from provenance logs and constructs a k-NN process similarity graph. In parallel, an Apriori-based rare pattern mining module discovers infrequent co-occurrence patterns and converts them into pattern evidence. A Graph Autoencoder is trained to learn normal relational structure. At inference, processes are scored using graph reconstruction residuals and then boosted with rare-pattern evidence to produce the final anomaly ranking.
• Figure 2: Example system provenance graph. Nodes represent system entities—processes (ovals), files (rectangles), and network endpoints (diamonds)—while directed edges encode causal interactions such as read, write, fork/execute, send, and receive. The dashed red boxes highlight subgraphs exhibiting suspicious behavior patterns (e.g., uncommon process spawning chains, sensitive file access, and outbound communication), illustrating the relational structure exploited by our rare-pattern-guided graph autoencoder for anomaly detection.
• Figure 3: PCA visualization of process behavioral feature space for Linux Dataset. We project all 272,376 processes, each described by 299 behavioral features, into two dimensions using Principal Component Analysis (PCA) for visualization. The first two components (PC1 and PC2) together capture approximately 52% of the total variance, highlighting the overall distribution of processes in the reduced space (with attack instances shown in red and normal instances in gray).