Structure-based Anomaly Detection and Clustering

TL;DR

This thesis proposes new unsupervised methods for anomaly detection in both structured and streaming data settings, and proposes Online Isolation Forest (Online-iForest), which uses adaptive, multi-resolution histograms and dynamically updates tree structures to track changes over time.

Abstract

Anomaly detection is a fundamental problem in domains such as healthcare, manufacturing, and cybersecurity. This thesis proposes new unsupervised methods for anomaly detection in both structured and streaming data settings. In the first part, we focus on structure-based anomaly detection, where normal data follows low-dimensional manifolds while anomalies deviate from them. We introduce Preference Isolation Forest (PIF), which embeds data into a high-dimensional preference space via manifold fitting, and isolates outliers using two variants: Voronoi-iForest, based on geometric distances, and RuzHash-iForest, leveraging Locality Sensitive Hashing for scalability. We also propose Sliding-PIF, which captures local manifold information for streaming scenarios. Our methods outperform existing techniques on synthetic and real datasets. We extend this to structure-based clustering with MultiLink, a novel method for recovering multiple geometric model families in noisy data. MultiLink merges clusters via a model-aware linkage strategy, enabling robust multi-class structure recovery. It offers key advantages over existing approaches, such as speed, reduced sensitivity to thresholds, and improved robustness to poor initial sampling. The second part of the thesis addresses online anomaly detection in evolving data streams. We propose Online Isolation Forest (Online-iForest), which uses adaptive, multi-resolution histograms and dynamically updates tree structures to track changes over time. It avoids retraining while achieving accuracy comparable to offline models, with superior efficiency for real-time applications. Finally, we tackle anomaly detection in cybersecurity via open-set recognition for malware classification. We enhance a Gradient Boosting classifier with MaxLogit to detect unseen malware families, a method now integrated into Cleafy's production system.

Figures (37)

• Figure 1: (a) An anomaly (marked as $\times$) is recognized as a point in a low density area. (b) Anomalies are defined with respect to their deviation from low-dimensional structures of given family $\mathcal{F}$, lines in this case.
• Figure 2: An example of structure-based clustering, where the goal is to recover genuine structures of multiple families $\mathcal{F}_1 \cup \dots \cup \mathcal{F}_k$, planes and cylinders in this case, from the input point cloud.
• Figure 3: The goal of online anomaly detection is to identify whether a new point $\textcolor{violet}{ \vb*{x} _t}$ comes from the genuine $\textcolor{green}{\Phi_0}$ or anomalous $\textcolor{red}{\Phi_1}$ distribution. A common challenge in the online scenario is the strict memory limitation, which only allows to store information from a limited time horizon $\textcolor{orange}{ \vb*{x} _{t-\omega}}, \dots, \textcolor{violet}{ \vb*{x} _t}$ for each time instant.
• Figure 4: Online-iForest is an ensemble of trees that continuously expand and contract their structure at each time instant $t$, by learning the new sample $\textcolor{violet}{ \vb*{x} _t}$, and forgetting the old sample $\textcolor{orange}{ \vb*{x} _{t-\omega}}$.
• Figure 5: The same genuine data $G$ can be described by different families $\mathcal{F}$ of parametric models, circles in (a) and lines in (b). (c) Unknown complex patterns can be described by models that locally approximate the underlying manifold.

Theorems & Definitions (7)

• Theorem 1
• Lemma 1: MinHash
• Theorem 1: RuzHash
• proof
• Theorem 2: $d_R( \vb*{p} , \vb*{q} )$ linearly correlates with $b$
• proof
• Corollary 1: $d_J( \vb*{p} , \vb*{q} )$ linearly correlates with $b$