Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Mutual Information Computation for Large Binary Datasets

Andre O. Falcao

TL;DR

This work addresses the computational bottleneck of calculating mutual information for large binary datasets by reworking MI computation into bulk matrix operations. It introduces a Gram-matrix based approach that computes all joint and marginal probabilities in a single pass, using a complementary data matrix to derive all required counts. The main contributions are a set of closed-form matrix expressions for all Gram matrices, a substantial reduction in computation time through vectorization, and thorough CPU-based benchmarking across dense and sparse implementations. The result is scalable, hardware-friendly MI computation suitable for high-dimensional binary data in genomics, NLP, and network analysis.

Abstract

Mutual Information (MI) is a powerful statistical measure that quantifies shared information between random variables, particularly valuable in high-dimensional data analysis across fields like genomics, natural language processing, and network science. However, computing MI becomes computationally prohibitive for large datasets where it is typically required a pairwise computational approach where each column is compared to others. This work introduces a matrix-based algorithm that accelerates MI computation by leveraging vectorized operations and optimized matrix calculations. By transforming traditional pairwise computational approaches into bulk matrix operations, the proposed method enables efficient MI calculation across all variable pairs. Experimental results demonstrate significant performance improvements, with computation times reduced up to 50,000 times in the largest dataset using optimized implementations, particularly when utilizing hardware optimized frameworks. The approach promises to expand MI's applicability in data-driven research by overcoming previous computational limitations.

Fast Mutual Information Computation for Large Binary Datasets

TL;DR

This work addresses the computational bottleneck of calculating mutual information for large binary datasets by reworking MI computation into bulk matrix operations. It introduces a Gram-matrix based approach that computes all joint and marginal probabilities in a single pass, using a complementary data matrix to derive all required counts. The main contributions are a set of closed-form matrix expressions for all Gram matrices, a substantial reduction in computation time through vectorization, and thorough CPU-based benchmarking across dense and sparse implementations. The result is scalable, hardware-friendly MI computation suitable for high-dimensional binary data in genomics, NLP, and network analysis.

Abstract

Mutual Information (MI) is a powerful statistical measure that quantifies shared information between random variables, particularly valuable in high-dimensional data analysis across fields like genomics, natural language processing, and network science. However, computing MI becomes computationally prohibitive for large datasets where it is typically required a pairwise computational approach where each column is compared to others. This work introduces a matrix-based algorithm that accelerates MI computation by leveraging vectorized operations and optimized matrix calculations. By transforming traditional pairwise computational approaches into bulk matrix operations, the proposed method enables efficient MI calculation across all variable pairs. Experimental results demonstrate significant performance improvements, with computation times reduced up to 50,000 times in the largest dataset using optimized implementations, particularly when utilizing hardware optimized frameworks. The approach promises to expand MI's applicability in data-driven research by overcoming previous computational limitations.

Paper Structure

This paper contains 17 sections, 26 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Core algorithm presentation
  3. Optimizing the computation of the Gram matrices
  4. Simplified computation of $\neg D$ Gram matrices
  5. Term 1: $\mathbf{1}^T \cdot \mathbf{1}$
  6. Term 2: $\mathbf{1}^T \cdot D$
  7. Term 3: $D^T \cdot \mathbf{1}$
  8. Term 4: $D^T \cdot D$
  9. Final Expression
  10. Algorithm complexity
  11. Implementation and results
  12. Experimental setup
  13. Results and discussion
  14. Global results
  15. Detailed implementation comparison
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Computation times of different algorithm implementations, changing the number of rows in the data set (number of columns fixed at 1000)
  • Figure 2: Computation times of different algorithm implementations, changing the number of rows in the data set (number of columns fixed at 1000)
  • Figure 3: Computation times (in seconds) across varying sparsity levels of the dataset (Simulated datasets with 100,000 rows and 1,000 columns)