Communication-Efficient and Memory-Aware Parallel Bootstrapping using MPI

TL;DR

The paper tackles the high cost of bootstrapping on large datasets in distributed MPI environments by proposing two memory-aware strategies. It introduces Local Statistic Aggregation (DBSA) to decouple communication from the number of resamples by transmitting two statistics and uses Var(X) = E[X^2]-E[X]^2; and Distributed Resampling with Synchronized RNG (DDRS) to enable resampling without storing the full dataset via deterministic RNG. Analytical models compare strategies A–D in terms of communication volume and memory usage, showing that DBSA achieves $O(D)$ communication and DDRS provides a memory-bounded option with $O(NP)$ communication. These results suggest scalable bootstrapping on large HPC systems, with practical implications for environments with limited memory or bandwidth.

Abstract

Bootstrapping is a powerful statistical resampling technique for estimating the sampling distribution of an estimator. However, its computational cost becomes prohibitive for large datasets or a high number of resamples. This paper presents a theoretical analysis and design of parallel bootstrapping algorithms using the Message Passing Interface (MPI). We address two key challenges: high communication overhead and memory constraints in distributed environments. We propose two novel strategies: 1) Local Statistic Aggregation, which drastically reduces communication by transmitting sufficient statistics instead of full resampled datasets, and 2) Synchronized Pseudo-Random Number Generation, which enables distributed resampling when the entire dataset cannot be stored on a single process. We develop analytical models for communication and computation complexity, comparing our methods against naive baseline approaches. Our analysis demonstrates that the proposed methods offer significant reductions in communication volume and memory usage, facilitating scalable parallel bootstrapping on large-scale systems.