Theoretical Investigation on Inductive Bias of Isolation Forest

TL;DR

The paper provides a theoretical foundation for the inductive bias of Isolation Forest by modeling iTree growth as a random walk and deriving a closed-form expected depth function $\bar{h}(x;D)$. It proves concentration of empirical depths to their theoretical values and conducts case studies contrasting iForest with $k$-NN across marginal, central, and clustered anomalies, revealing that iForest is less sensitive to central anomalies and more parameter-adaptive. The work offers a principled explanation for iForest's effectiveness, establishes a framework to study its behavior, and discusses challenges and future directions for multi-dimensional analysis. These insights enable deeper theoretical understanding and guide practical use without extensive hyperparameter tuning.

Abstract

Isolation Forest (iForest) stands out as a widely-used unsupervised anomaly detector, primarily owing to its remarkable runtime efficiency and superior performance in large-scale tasks. Despite its widespread adoption, a theoretical foundation explaining iForest's success remains unclear. This paper focuses on the inductive bias of iForest, which theoretically elucidates under what circumstances and to what extent iForest works well. The key is to formulate the growth process of iForest, where the split dimensions and split values are randomly selected. We model the growth process of iForest as a random walk, enabling us to derive the expected depth function, which is the outcome of iForest, using transition probabilities. The case studies reveal key inductive biases: iForest exhibits lower sensitivity to central anomalies while demonstrating greater parameter adaptability compared to $k$-Nearest Neighbor. Our study provides a theoretical understanding of the effectiveness of iForest and establishes a foundation for further theoretical exploration.

Figures (8)

• Figure 1: Examples of the random walk model for iTrees. The points colored in black, blue, and red indicate the initial, intermediate, and absorbing states, respectively. The lines with arrows indicate the trajectory of the transition process.
• Figure 2: Marginal single anomaly.
• Figure 3: Central single anomaly.
• Figure 4: Marginal multiple anomalies.
• Figure 5: Mean-square errors of the real depths learned by iForest to the theoretical expected depths about numbers of trees. The shaded regions represent the confidence regions over multiple runs.

Theorems & Definitions (18)

• Proposition 3.1: Concentration of iForest
• Theorem 3.2: Random Walk Model for iTree
• Lemma 3.3
• Lemma 3.4
• Theorem 3.5
• Definition 4.1
• Theorem 4.3
• Theorem 4.4
• Theorem 4.5
• Theorem 4.6