Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Source Coding for Parametric and Non-Parametric Regression

Jiahui Wei, Elsa Dupraz, Philippe Mary

TL;DR

This work addresses regression under goal-oriented communication constraints by formulating distributed source coding with side information and analyzing both parametric and kernel regression. It introduces a Gaussian test-channel coding scheme based on the Draper universal approach to obtain rate-generalization error regions in both asymptotic and finite block-length regimes. The authors prove that, asymptotically, there is no trade-off between data reconstruction and regression, and they extend the finite-length analysis with dispersion-based tools, demonstrating decorrelation between distortion and generalization error under the proposed scheme. Numerical results corroborate the theoretical findings, showing convergence to the asymptotic region and no finite-length trade-off, with parametric regression outperforming kernel methods in convergence speed. The results advance goal-oriented communications by guiding the design of distributed coding schemes that preserve learning performance under rate constraints for regression tasks, including both parametric and non-parametric settings.

Abstract

The design of communication systems dedicated to machine learning tasks is one key aspect of goal-oriented communications. In this framework, this article investigates the interplay between data reconstruction and learning from the same compressed observations, particularly focusing on the regression problem. We establish achievable rate-generalization error regions for both parametric and non-parametric regression, where the generalization error measures the regression performance on previously unseen data. The analysis covers both asymptotic and finite block-length regimes, providing fundamental results and practical insights for the design of coding schemes dedicated to regression. The asymptotic analysis relies on conventional Wyner-Ziv coding schemes which we extend to study the convergence of the generalization error. The finite-length analysis uses the notions of information density and dispersion with additional term for the generalization error. We further investigate the trade-off between reconstruction and regression in both asymptotic and non-asymptotic regimes. Contrary to the existing literature which focused on other learning tasks, our results state that in the case of regression, there is no trade-off between data reconstruction and regression in the asymptotic regime. We also observe the same absence of trade-off for the considered achievable scheme in the finite-length regime, by analyzing correlation between distortion and generalization error.

Distributed Source Coding for Parametric and Non-Parametric Regression

TL;DR

This work addresses regression under goal-oriented communication constraints by formulating distributed source coding with side information and analyzing both parametric and kernel regression. It introduces a Gaussian test-channel coding scheme based on the Draper universal approach to obtain rate-generalization error regions in both asymptotic and finite block-length regimes. The authors prove that, asymptotically, there is no trade-off between data reconstruction and regression, and they extend the finite-length analysis with dispersion-based tools, demonstrating decorrelation between distortion and generalization error under the proposed scheme. Numerical results corroborate the theoretical findings, showing convergence to the asymptotic region and no finite-length trade-off, with parametric regression outperforming kernel methods in convergence speed. The results advance goal-oriented communications by guiding the design of distributed coding schemes that preserve learning performance under rate constraints for regression tasks, including both parametric and non-parametric settings.

Abstract

The design of communication systems dedicated to machine learning tasks is one key aspect of goal-oriented communications. In this framework, this article investigates the interplay between data reconstruction and learning from the same compressed observations, particularly focusing on the regression problem. We establish achievable rate-generalization error regions for both parametric and non-parametric regression, where the generalization error measures the regression performance on previously unseen data. The analysis covers both asymptotic and finite block-length regimes, providing fundamental results and practical insights for the design of coding schemes dedicated to regression. The asymptotic analysis relies on conventional Wyner-Ziv coding schemes which we extend to study the convergence of the generalization error. The finite-length analysis uses the notions of information density and dispersion with additional term for the generalization error. We further investigate the trade-off between reconstruction and regression in both asymptotic and non-asymptotic regimes. Contrary to the existing literature which focused on other learning tasks, our results state that in the case of regression, there is no trade-off between data reconstruction and regression in the asymptotic regime. We also observe the same absence of trade-off for the considered achievable scheme in the finite-length regime, by analyzing correlation between distortion and generalization error.
Paper Structure (37 sections, 9 theorems, 101 equations, 2 figures)

This paper contains 37 sections, 9 theorems, 101 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Context and problem
  3. Related works
  4. Contributions
  5. Regression problem
  6. Notation
  7. Source definitions
  8. Parametric regression
  9. Non-parametric regression
  10. Coding scheme for regression
  11. Training and inference phases
  12. Generalization error
  13. Coding scheme
  14. Asymptotic Analysis
  15. Achievable rate-generalization error regions
  16. ...and 22 more sections

Key Result

Theorem 1

Given any rate $R > 0$, the pair $(R, 0)$ is achievable for parametric regression with quadratic loss, for sources $(X,Y)$ following the model in para_reg.

Figures (2)

  • Figure 1: Coding scheme for regression, with one training phase (a) over the learning sequence $\mathbf{Z}=(\mathbf{U},\mathbf{Y})$ which provides a predictor $\hat{f}^{(n)}(\mathbf{Z},.)$, and one inference phase (b) which consists of applying the predictor on new samples $\tilde{Y}$.
  • Figure 2: Non-asymptotic rate-distortion-generalization error region

Theorems & Definitions (21)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8
  • Theorem 1: Parametric regression
  • Theorem 2: Kernel regression
  • ...and 11 more