Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Tree Ensemble Proximities in Python

Adrien Aumon, Guy Wolf, Kevin R. Moon, Jake S. Rhodes

TL;DR

Tree ensemble proximities enable supervised similarity measures but traditionally suffer from quadratic time and memory costs. The authors introduce Separable Weighted Leaf-Collision Proximities (WLCP) that admit an exact sparse factorization $P = QW^T$, reducing computation to leaf collisions and enabling near-linear time/space on standard CPUs. This framework generalizes several known proximities (e.g., RF, RF-GAP, GBT) and is implemented in Python using sparse linear algebra, dramatically improving scalability. Empirical results on large-scale datasets demonstrate substantial speedups and memory savings over prior approaches, validating the method's practicality for proximity-based analysis at scale.

Abstract

Tree ensemble methods such as Random Forests naturally induce supervised similarity measures through their decision tree structure, but existing implementations of proximities derived from tree ensembles typically suffer from quadratic time or memory complexity, limiting their scalability. In this work, we introduce a general framework for efficient proximity computation by defining a family of Separable Weighted Leaf-Collision Proximities. We show that any proximity measure in this family admits an exact sparse matrix factorization, restricting computation to leaf-level collisions and avoiding explicit pairwise comparisons. This formulation enables low-memory, scalable proximity computation using sparse linear algebra in Python. Empirical benchmarks demonstrate substantial runtime and memory improvements over traditional approaches, allowing tree ensemble proximities to scale efficiently to datasets with hundreds of thousands of samples on standard CPU hardware.

Scalable Tree Ensemble Proximities in Python

TL;DR

Tree ensemble proximities enable supervised similarity measures but traditionally suffer from quadratic time and memory costs. The authors introduce Separable Weighted Leaf-Collision Proximities (WLCP) that admit an exact sparse factorization , reducing computation to leaf collisions and enabling near-linear time/space on standard CPUs. This framework generalizes several known proximities (e.g., RF, RF-GAP, GBT) and is implemented in Python using sparse linear algebra, dramatically improving scalability. Empirical results on large-scale datasets demonstrate substantial speedups and memory savings over prior approaches, validating the method's practicality for proximity-based analysis at scale.

Abstract

Tree ensemble methods such as Random Forests naturally induce supervised similarity measures through their decision tree structure, but existing implementations of proximities derived from tree ensembles typically suffer from quadratic time or memory complexity, limiting their scalability. In this work, we introduce a general framework for efficient proximity computation by defining a family of Separable Weighted Leaf-Collision Proximities. We show that any proximity measure in this family admits an exact sparse matrix factorization, restricting computation to leaf-level collisions and avoiding explicit pairwise comparisons. This formulation enables low-memory, scalable proximity computation using sparse linear algebra in Python. Empirical benchmarks demonstrate substantial runtime and memory improvements over traditional approaches, allowing tree ensemble proximities to scale efficiently to datasets with hundreds of thousands of samples on standard CPU hardware.
Paper Structure (9 sections, 2 theorems, 6 equations, 1 figure)

This paper contains 9 sections, 2 theorems, 6 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Methods
  3. Factorizable leaf-collision proximities
  4. Complexity analysis
  5. Time complexity.
  6. Space complexity.
  7. Benchmark
  8. Conclusion
  9. Proofs

Key Result

Proposition 1

Any separable WLCP matrix $P$ can be exactly factorized as $P = QW^\top$, where $Q, W \in \mathbb{R}^{N \times |\mathcal{K}|}$ and $\mathcal{K}$ is the set of all unique leaves in the ensemble of trees. Furthermore, $Q$ and $W$ are strictly row-sparse matrices with at most$T$ non-zero entries per ro

Figures (1)

  • Figure 1: Empirical complexity and scalability analysis on the Fashion-MNIST dataset comparing naive pairwise implementations with our sparse framework. (a) Runtime analysis demonstrates near-linear scaling ($m \approx 1.1$) for the sparse method, while pairwise implementations exhibit quadratic growth ($m \approx 2.0$). (b) Memory usage scaling highlights the efficiency of the sparse approach; pairwise curves are truncated at $N=50,000$ and $N=60,000$ due to process termination resulting from memory exhaustion.

Theorems & Definitions (3)

  • Definition 1: Weighted Leaf-Collision Proximities
  • Proposition 1: Sparse matrix factorization
  • Corollary 1.1