Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed and Rate-Adaptive Feature Compression

Aditya Deshmukh, Venugopal V. Veeravalli, Gunjan Verma

TL;DR

This work tackles distributed feature compression under variable communication rates for inference using pretrained models. For linear regressors, it proves that optimal distributed quantizers align with a one-dimensional projection of sensor data, enabling a simple, rate-adaptive scheme based on 1D clustering; it also introduces an adaptive mechanism that reduces bitrate without re-training. To extend to general models, the authors deploy a VQ-VAE framework that encodes sensor observations into low-dimensional latent codes, with end-to-end training fixed to the pretrained model and an adaptive scheme that reuses codebooks via weighted clustering. Experiments on synthetic data and benchmarks (MNIST Audio+Image, CIFAR-10) show the adaptive methods closely match or surpass non-adaptive baselines while substantially reducing communication load, highlighting practical impact for latency-sensitive, bandwidth-constrained deployments.

Abstract

We study the problem of distributed and rate-adaptive feature compression for linear regression. A set of distributed sensors collect disjoint features of regressor data. A fusion center is assumed to contain a pretrained linear regression model, trained on a dataset of the entire uncompressed data. At inference time, the sensors compress their observations and send them to the fusion center through communication-constrained channels, whose rates can change with time. Our goal is to design a feature compression {scheme} that can adapt to the varying communication constraints, while maximizing the inference performance at the fusion center. We first obtain the form of optimal quantizers assuming knowledge of underlying regressor data distribution. Under a practically reasonable approximation, we then propose a distributed compression scheme which works by quantizing a one-dimensional projection of the sensor data. We also propose a simple adaptive scheme for handling changes in communication constraints. We demonstrate the effectiveness of the distributed adaptive compression scheme through simulated experiments.

Distributed and Rate-Adaptive Feature Compression

TL;DR

This work tackles distributed feature compression under variable communication rates for inference using pretrained models. For linear regressors, it proves that optimal distributed quantizers align with a one-dimensional projection of sensor data, enabling a simple, rate-adaptive scheme based on 1D clustering; it also introduces an adaptive mechanism that reduces bitrate without re-training. To extend to general models, the authors deploy a VQ-VAE framework that encodes sensor observations into low-dimensional latent codes, with end-to-end training fixed to the pretrained model and an adaptive scheme that reuses codebooks via weighted clustering. Experiments on synthetic data and benchmarks (MNIST Audio+Image, CIFAR-10) show the adaptive methods closely match or surpass non-adaptive baselines while substantially reducing communication load, highlighting practical impact for latency-sensitive, bandwidth-constrained deployments.

Abstract

We study the problem of distributed and rate-adaptive feature compression for linear regression. A set of distributed sensors collect disjoint features of regressor data. A fusion center is assumed to contain a pretrained linear regression model, trained on a dataset of the entire uncompressed data. At inference time, the sensors compress their observations and send them to the fusion center through communication-constrained channels, whose rates can change with time. Our goal is to design a feature compression {scheme} that can adapt to the varying communication constraints, while maximizing the inference performance at the fusion center. We first obtain the form of optimal quantizers assuming knowledge of underlying regressor data distribution. Under a practically reasonable approximation, we then propose a distributed compression scheme which works by quantizing a one-dimensional projection of the sensor data. We also propose a simple adaptive scheme for handling changes in communication constraints. We demonstrate the effectiveness of the distributed adaptive compression scheme through simulated experiments.
Paper Structure (10 sections, 1 theorem, 26 equations, 6 figures, 2 algorithms)

This paper contains 10 sections, 1 theorem, 26 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Adaptive Quantization
  4. Experimental Results
  5. General Learning Model
  6. Adaptive Compression
  7. Experiments
  8. MNIST Audio and Image
  9. CIFAR-10
  10. Conclusion

Key Result

Lemma 1

The optimal quantizers for optimization problem in eq:obj_linreg simultaneously satisfy the following structure (which is provided for an arbitrary $i$th sensor): where $\bar{c}_k^{(i)}$ and $\bar{\mathcal{D}}_k^{(i)}$ are the minimizers of where $\mu_i$ is the Lebesgue measure on $\mathbb{R}^{d_i}$, $p_i$ is the probability density function of $\bm x^{(i)}$ with respect to $\mu_i$ and Note tha

Figures (6)

  • Figure 1: Distributed system with multi-modality
  • Figure 2: MSE values obtained by the model-agnostic, non-adaptive, and proposed adaptive strategies.
  • Figure 3: VQ-VAE based distributed system
  • Figure 4: DNN architecture for MNIST Audio+Image experiment
  • Figure 5: Performance of the non-adaptive and proposed adaptive strategies on the MNIST Audio+Image dataset.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Remark 1
  • Lemma 1
  • proof
  • Remark 2