Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Learning of Multivariate Distributions via Coresets

Zeyu Ding, Katja Ickstadt, Nadja Klein, Alexander Munteanu, Simon Omlor

Abstract

Efficient and scalable non-parametric or semi-parametric regression analysis and density estimation are of crucial importance to the fields of statistics and machine learning. However, available methods are limited in their ability to handle large-scale data. We address this issue by developing a novel coreset construction for multivariate conditional transformation models (MCTMs) to enhance their scalability and training efficiency. To the best of our knowledge, these are the first coresets for semi-parametric distributional models. Our approach yields substantial data reduction via importance sampling. It ensures with high probability that the log-likelihood remains within multiplicative error bounds of $(1\pm\varepsilon)$ and thereby maintains statistical model accuracy. Compared to conventional full-parametric models, where coresets have been incorporated before, our semi-parametric approach exhibits enhanced adaptability, particularly in scenarios where complex distributions and non-linear relationships are present, but not fully understood. To address numerical problems associated with normalizing logarithmic terms, we follow a geometric approximation based on the convex hull of input data. This ensures feasible, stable, and accurate inference in scenarios involving large amounts of data. Numerical experiments demonstrate substantially improved computational efficiency when handling large and complex datasets, thus laying the foundation for a broad range of applications within the statistics and machine learning communities.

Scalable Learning of Multivariate Distributions via Coresets

Abstract

Efficient and scalable non-parametric or semi-parametric regression analysis and density estimation are of crucial importance to the fields of statistics and machine learning. However, available methods are limited in their ability to handle large-scale data. We address this issue by developing a novel coreset construction for multivariate conditional transformation models (MCTMs) to enhance their scalability and training efficiency. To the best of our knowledge, these are the first coresets for semi-parametric distributional models. Our approach yields substantial data reduction via importance sampling. It ensures with high probability that the log-likelihood remains within multiplicative error bounds of and thereby maintains statistical model accuracy. Compared to conventional full-parametric models, where coresets have been incorporated before, our semi-parametric approach exhibits enhanced adaptability, particularly in scenarios where complex distributions and non-linear relationships are present, but not fully understood. To address numerical problems associated with normalizing logarithmic terms, we follow a geometric approximation based on the convex hull of input data. This ensures feasible, stable, and accurate inference in scenarios involving large amounts of data. Numerical experiments demonstrate substantially improved computational efficiency when handling large and complex datasets, thus laying the foundation for a broad range of applications within the statistics and machine learning communities.
Paper Structure (30 sections, 14 theorems, 58 equations, 13 figures, 8 tables, 2 algorithms)

This paper contains 30 sections, 14 theorems, 58 equations, 13 figures, 8 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Brief Introduction to MCTMs
  3. Our Goals and Contributions
  4. Related Work
  5. MCTM CORESET CONSTRUCTION
  6. EMPIRICAL EVALUATION
  7. Simulation Study on 2-Dimensional Data
  8. Real-World Data Experiments
  9. DISCUSSION AND EXTENSIONS
  10. SUMMARY AND CONCLUSION
  11. THEORETICAL RESULTS
  12. Preliminaries
  13. Squared Part
  14. Logarithmic Parts
  15. Main Result
  16. ...and 15 more sections

Key Result

Lemma 2.0

There exists a coreset $S$ for $f_1$ of size $O(J^2 d /\varepsilon^2)$ which can be computed in time $O(\mathrm{nnz}(B)\log(nJ) +\mathrm{poly}(dJ))$ and with high probability by sampling and reweighting according to the $\ell_2$ leverage scores of $B$, where $\mathrm{nnz}(B)$ denotes the number of n

Figures (13)

  • Figure 1: Coreset performance comparison on stock-return data. Top row: 10 stocks; bottom row: 20 stocks. Shaded bands indicate $\pm 1$ standard deviation, and solid lines show the averages over multiple repetitions.
  • Figure 2: Coreset visualization for basic probability distributions. Each row shows a different sampling method: Uniform (top), $\ell_2$ Sensitivity (middle), and $\ell_2$-Hull (bottom).
  • Figure 3: Coreset visualization for complex probability distributions. Each column shows a different sampling method: Uniform (top), $\ell_2$ Sensitivity (middle), and $\ell_2$-Hull (bottom).
  • Figure 4: Coreset visualization for geometric dependency structures. Each column shows a different sampling method: Uniform (top), $\ell_2$ Sensitivity (middle), and $\ell_2$-Hull (bottom).
  • Figure 5: Coreset visualization for additional dependency structures. Each column shows a different sampling method: Uniform (top), $\ell_2$ Sensitivity (middle), and $\ell_2$-Hull (bottom).
  • ...and 8 more figures

Theorems & Definitions (22)

  • Lemma 2.0
  • Lemma 2.0
  • Lemma 2.0
  • Theorem 2.1
  • Lemma 2.1
  • Lemma 2.1
  • Lemma A.0
  • proof
  • Lemma A.0
  • proof
  • ...and 12 more