Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Biased Estimator for MinMax Sampling and Distributed Aggregation

Joel Wolfrath, Abhishek Chandra

TL;DR

This paper tackles reducing communication for downsampling real-valued vectors in geo-distributed data using MinMax sampling. It introduces B-MinMax, a biased estimator that sends raw components with probability p_{i,j} rather than unbiased scaled values, trading bias for reduced variance. The authors derive the MSE behavior for single-site and aggregated settings and propose an adaptive mechanism that estimates MSE using per-site statistics to decide between biased and unbiased aggregation. Empirical results on Zipfian data and ResNet-18 weights show substantial MSE reductions at high compression and small-scale aggregation, with a mechanism to defer to unbiased estimation when aggregation is large. The work has practical impact for WAN-limited distributed analytics and federated settings where communication is costly.

Abstract

MinMax sampling is a technique for downsampling a real-valued vector which minimizes the maximum variance over all vector components. This approach is useful for reducing the amount of data that must be sent over a constrained network link (e.g. in the wide-area). MinMax can provide unbiased estimates of the vector elements, along with unbiased estimates of aggregates when vectors are combined from multiple locations. In this work, we propose a biased MinMax estimation scheme, B-MinMax, which trades an increase in estimator bias for a reduction in variance. We prove that when no aggregation is performed, B-MinMax obtains a strictly lower MSE compared to the unbiased MinMax estimator. When aggregation is required, B-MinMax is preferable when sample sizes are small or the number of aggregated vectors is limited. Our experiments show that this approach can substantially reduce the MSE for MinMax sampling in many practical settings.

A Biased Estimator for MinMax Sampling and Distributed Aggregation

TL;DR

This paper tackles reducing communication for downsampling real-valued vectors in geo-distributed data using MinMax sampling. It introduces B-MinMax, a biased estimator that sends raw components with probability p_{i,j} rather than unbiased scaled values, trading bias for reduced variance. The authors derive the MSE behavior for single-site and aggregated settings and propose an adaptive mechanism that estimates MSE using per-site statistics to decide between biased and unbiased aggregation. Empirical results on Zipfian data and ResNet-18 weights show substantial MSE reductions at high compression and small-scale aggregation, with a mechanism to defer to unbiased estimation when aggregation is large. The work has practical impact for WAN-limited distributed analytics and federated settings where communication is costly.

Abstract

MinMax sampling is a technique for downsampling a real-valued vector which minimizes the maximum variance over all vector components. This approach is useful for reducing the amount of data that must be sent over a constrained network link (e.g. in the wide-area). MinMax can provide unbiased estimates of the vector elements, along with unbiased estimates of aggregates when vectors are combined from multiple locations. In this work, we propose a biased MinMax estimation scheme, B-MinMax, which trades an increase in estimator bias for a reduction in variance. We prove that when no aggregation is performed, B-MinMax obtains a strictly lower MSE compared to the unbiased MinMax estimator. When aggregation is required, B-MinMax is preferable when sample sizes are small or the number of aggregated vectors is limited. Our experiments show that this approach can substantially reduce the MSE for MinMax sampling in many practical settings.
Paper Structure (10 sections, 2 theorems, 17 equations, 4 figures)

This paper contains 10 sections, 2 theorems, 17 equations, 4 figures.

Table of Contents

  1. Introduction
  2. MinMax Estimation
  3. The B-MinMax Estimator
  4. B-MinMax without Aggregation
  5. B-MinMax with Aggregation
  6. Empirical Evaluation
  7. Compression Ratio
  8. Number of Sites
  9. Discussion
  10. Conclusion

Key Result

Theorem 3.1

Figures (4)

  • Figure 1: Overview of B-MinMax Sampling with $k=2$ sites and a sample size of $n_1 = n_2 = 2$.
  • Figure 2: MSE using aggregation across multiple sites, with ${x}_{i,j} \sim Zipf(1)$
  • Figure 3: MSE across different compression ratios, with $k=4$.
  • Figure 4: MSE across different numbers of sites, with a compression ratio of 4x.

Theorems & Definitions (4)

  • Theorem 3.1
  • proof
  • Theorem 3.4
  • proof