RFX: High-Performance Random Forests with GPU Acceleration and QLORA Compression

TL;DR

RFX addresses the proximity memory bottleneck in Breiman–Cutler Random Forests by combining graphical-method fidelity with modern memory- and compute-optimized techniques. The core approach deploys QLORA proximity compression and CPU TriBlock proximity to drastically reduce proximity-matrix memory, alongside GPU-accelerated tree growth, importance computation, and 3D MDS visualization from low-rank factors. The key contributions are a production-ready, open-source RF classifier that preserves Breiman–Cutler methodology, achieves 100k+ sample proximity analysis on commodity hardware, and delivers interactive visualization (rfviz) for case-wise and cluster-aware interpretability. The practical impact is enabling proximity-based analysis at scales previously infeasible, with demonstrated gains in speed and memory efficiency while maintaining near-lossless quality for visualization and downstream tasks.

Abstract

RFX (Random Forests X), where X stands for compression or quantization, presents a production-ready implementation of Breiman and Cutler's Random Forest classification methodology in Python. RFX v1.0 provides complete classification: out-of-bag error estimation, overall and local importance measures, proximity matrices with QLORA compression, case-wise analysis, and interactive visualization (rfviz)--all with CPU and GPU acceleration. Regression, unsupervised learning, CLIQUE importance, and RF-GAP proximity are planned for v2.0. This work introduces four solutions addressing the proximity matrix memory bottleneck limiting Random Forest analysis to ~60,000 samples: (1) QLORA (Quantized Low-Rank Adaptation) compression for GPU proximity matrices, reducing memory from 80GB to 6.4MB for 100k samples (12,500x compression with INT8 quantization) while maintaining 99% geometric structure preservation, (2) CPU TriBlock proximity--combining upper-triangle storage with block-sparse thresholding--achieving 2.7x memory reduction with lossless quality, (3) SM-aware GPU batch sizing achieving 95% GPU utilization, and (4) GPU-accelerated 3D MDS visualization computing embeddings directly from low-rank factors using power iteration. Validation across four implementation modes (GPU/CPU x case-wise/non-case-wise) demonstrates correct implementation. GPU achieves 1.4x speedup over CPU for overall importance with 500+ trees. Proximity computation scales from 1,000 to 200,000+ samples (requiring GPU QLORA), with CPU TriBlock filling the gap for medium-scale datasets (10K-50K samples). RFX v1.0 eliminates the proximity memory bottleneck, enabling proximity-based Random Forest analysis on datasets orders of magnitude larger than previously feasible. Open-source production-ready classification following Breiman and Cutler's original methodology.

Figures (4)

• Figure 1: 3D MDS proximity embedding: samples positioned based on Random Forest proximity with interactive rotation and selection.
• Figure 2: Parallel coordinates plot: each line represents a sample's local importance values across features, revealing patterns through coordinated trajectories.
• Figure 3: Visual comparison of case-wise vs. non-case-wise analysis modes on Wine dataset (178 samples, 13 features, 3 classes, 500 trees). Top row: 3D MDS embeddings from proximity matrices showing cluster structure. Bottom row: Parallel coordinate plots of local importance across features, demonstrating sample-specific importance patterns (case-wise) vs. global patterns (non-case-wise). Case-wise mode enables finer-grained interpretability at the individual sample level.
• Figure 4: RFviz 2×2 grid from rf.rfviz() on Wine dataset (178 samples, 13 features, 3 classes). Linked brushing enables coordinated exploration across input features (top-left), 3D MDS proximity (top-right), local importance (bottom-left), and RAFT class votes heatmap (bottom-right).