NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data

TL;DR

NApy targets the bottleneck of computing pairwise statistics on large heterogeneous datasets with missing values in Python by offering a C++ OpenMP backend with a Python wrapper and a complementary Numba path. It implements seven tests across continuous, dichotomous, and categorical features, performing on-the-fly pairwise missing-data removal and providing test statistics, $P$-values (with multiple-testing corrections), and effect sizes. Benchmark results on simulated data and the CHRIS cohort show orders-of-magnitude improvements in runtime and memory efficiency over SciPy, Pingouin, and pandas baselines, enabling real-time interactive exploration of association networks. The work provides open-source code and benchmarking data to facilitate adoption in interactive biomedical analytics and large-scale data exploration.

Abstract

Existing Python libraries and tools lack the ability to efficiently compute statistical test results for large datasets in the presence of missing values. This presents an issue as soon as constraints on runtime and memory availability become essential considerations for a particular usecase. Relevant research areas where such limitations arise include interactive tools and databases for exploratory analysis of biomedical data. To address this problem, we present the Python package NApy, which relies on a Numba and C++ backend with OpenMP parallelization to enable scalable statistical testing for mixed-type datasets in the presence of missing values. Both with respect to runtime and memory consumption, NApy outperforms competitor tools and baseline implementations with naive Python-based parallelization by orders of magnitude, thereby enabling on-the-fly analyses in interactive applications. NApy is publicly available at https://github.com/DyHealthNet/NApy.

Figures (5)

• Figure 1: Distribution of missing values in the CHRIS study for continuous, dichotomous, and categorical features.
• Figure 2: Runtime evaluation of NApy's Numba and C++ implementation of the statistical tests performed using varying numbers of threads on datasets comprising 1000 samples and features with 10% of missing values per feature. The log10 fold change between the C++ and Numba implementations was computed on the average runtime of three independent runs per statistical test and thread count.
• Figure 3: Runtime benchmarks of NApy and applicable competitors across features, sample, and thread counts. (A--C) The log10 fold changes between the runtimes of the fastest competitor (pandas for correlations and Scipy-Python Loop for the remaining tests) and NApy were calculated across the different configurations of features (A), samples (B), and threads (C). (D) The runtime (in seconds) was measured on a single thread for datasets with 1000 samples and varying number of features. (E) Datasets with 1000 features were used to investigate the sample effect on runtime, with computations being performed on a single thread. (F) Datasets of 1000 samples and features were simulated and statistical tests were performed on varying number of threads. As pandas does not support intrinsic parallelization, the dashed line represents its runtime on a single thread. All datasets included 10% of missing values per feature, and the runtimes were averaged over three independent simulation runs, with standard deviations displayed as error bars.
• Figure 4: Benchmark analysis of the impact of missing values per feature on the runtime of statistical tests in NApy. Simulated datasets with fixed sizes of 1000 features and 1000 samples were analyzed to evaluate the effect of varying levels of missing values per feature on runtime. All computations were performed using a single thread, and runtime measurements were averaged over three runs, with standard deviations represented as error bars.
• Figure 5: Runtime evaluation of NApy and its fastest competitor on the CHRIS study data. The comparison includes pandas for correlation tests and Scipy-Python Loop for other statistical tests. The log10 fold change was derived from the average runtime of three independent runs per statistical test and thread count.