Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Secure Sparse Matrix Multiplications and their Applications to Privacy-Preserving Machine Learning

Marc Damie, Florian Hahn, Andreas Peter, Jan Ramon

TL;DR

This work advances privacy-preserving machine learning by designing secure sparse matrix multiplications within an outsourced MPC framework, addressing memory and communication bottlenecks that plague dense secure methods. It introduces two sparse-multiplication algorithms built on oblivious sorting/shuffling to efficiently handle secret-shared sparse data, achieving substantial performance gains and enabling PPML tasks (e.g., recommender systems, access control) that are impractical with prior protocols. The authors analyze the necessary public knowledge about sparsity, propose techniques to minimize and privately estimate this information (row anonymization, max-row padding, and matrix templating), and demonstrate memory and communication reductions up to about two orders of magnitude in realistic scenarios. The work thus broadens the applicability of PPML by making secure sparse computations feasible at scale while maintaining rigorous privacy guarantees, and provides open-source implementations for integration into existing MPC frameworks.

Abstract

To preserve privacy, multi-party computation (MPC) enables executing Machine Learning (ML) algorithms on secret-shared or encrypted data. However, existing MPC frameworks are not optimized for sparse data. This makes them unsuitable for ML applications involving sparse data, e.g., recommender systems or genomics. Even in plaintext, such applications involve high-dimensional sparse data, that cannot be processed without sparsity-related optimizations due to prohibitively large memory requirements. Since matrix multiplication is central in ML algorithms, we propose MPC algorithms to multiply secret sparse matrices. On the one hand, our algorithms avoid the memory issues of the "dense" data representation of classic secure matrix multiplication algorithms. On the other hand, our algorithms can significantly reduce communication costs (some experiments show a factor 1000) for realistic problem sizes. We validate our algorithms in two ML applications in which existing protocols are impractical. An important question when developing MPC algorithms is what assumptions can be made. In our case, if the number of non-zeros in a row is a sensitive piece of information then a short runtime may reveal that the number of non-zeros is small. Existing approaches make relatively simple assumptions, e.g., that there is a universal upper bound to the number of non-zeros in a row. This often doesn't align with statistical reality, in a lot of sparse datasets the amount of data per instance satisfies a power law. We propose an approach which allows adopting a safe upper bound on the distribution of non-zeros in rows/columns of sparse matrices.

Secure Sparse Matrix Multiplications and their Applications to Privacy-Preserving Machine Learning

TL;DR

This work advances privacy-preserving machine learning by designing secure sparse matrix multiplications within an outsourced MPC framework, addressing memory and communication bottlenecks that plague dense secure methods. It introduces two sparse-multiplication algorithms built on oblivious sorting/shuffling to efficiently handle secret-shared sparse data, achieving substantial performance gains and enabling PPML tasks (e.g., recommender systems, access control) that are impractical with prior protocols. The authors analyze the necessary public knowledge about sparsity, propose techniques to minimize and privately estimate this information (row anonymization, max-row padding, and matrix templating), and demonstrate memory and communication reductions up to about two orders of magnitude in realistic scenarios. The work thus broadens the applicability of PPML by making secure sparse computations feasible at scale while maintaining rigorous privacy guarantees, and provides open-source implementations for integration into existing MPC frameworks.

Abstract

To preserve privacy, multi-party computation (MPC) enables executing Machine Learning (ML) algorithms on secret-shared or encrypted data. However, existing MPC frameworks are not optimized for sparse data. This makes them unsuitable for ML applications involving sparse data, e.g., recommender systems or genomics. Even in plaintext, such applications involve high-dimensional sparse data, that cannot be processed without sparsity-related optimizations due to prohibitively large memory requirements. Since matrix multiplication is central in ML algorithms, we propose MPC algorithms to multiply secret sparse matrices. On the one hand, our algorithms avoid the memory issues of the "dense" data representation of classic secure matrix multiplication algorithms. On the other hand, our algorithms can significantly reduce communication costs (some experiments show a factor 1000) for realistic problem sizes. We validate our algorithms in two ML applications in which existing protocols are impractical. An important question when developing MPC algorithms is what assumptions can be made. In our case, if the number of non-zeros in a row is a sensitive piece of information then a short runtime may reveal that the number of non-zeros is small. Existing approaches make relatively simple assumptions, e.g., that there is a universal upper bound to the number of non-zeros in a row. This often doesn't align with statistical reality, in a lot of sparse datasets the amount of data per instance satisfies a power law. We propose an approach which allows adopting a safe upper bound on the distribution of non-zeros in rows/columns of sparse matrices.
Paper Structure (49 sections, 2 theorems, 8 equations, 6 figures, 2 tables, 10 algorithms)

This paper contains 49 sections, 2 theorems, 8 equations, 6 figures, 2 tables, 10 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Our contributions
  4. Notations
  5. Preliminaries
  6. Threat model
  7. Outsourced setting
  8. MPC primitives
  9. Matrix multiplication
  10. Dense matrix multiplication on secret-shared data
  11. Dishonest majority
  12. Honest majority
  13. Sparse data in PPML
  14. Sparse data representation
  15. Larger datasets are sparser
  16. ...and 34 more sections

Key Result

Lemma 1

$\mathcal{M}^+_{\epsilon,\delta}$ is $\epsilon$-DP and with probability $1-\delta$ there holds $y \le \mathcal{M}^+_{\epsilon,\delta}(y)$.

Figures (6)

  • Figure 1: Outsourced MPC setting
  • Figure 2: Comparison of the secure matrix multiplications under honest majority: dense algorithm vs. our sparse algorithm with three sparsity levels.
  • Figure 3: Per-row sparsity distribution in Bookcrossing ziegler_improving_2005
  • Figure 4: Matrix storage costs for different techniques of public knowledge minimization on four real-world datasets
  • Figure 5: Distribution of row non-zero counts, and DP safe upper bounds for variable template block sizes
  • ...and 1 more figures

Theorems & Definitions (4)

  • Lemma 1
  • proof
  • Lemma 2
  • proof