Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient l_0 Penalized Least Square

Chenqi Gong, Hu Yang

TL;DR

This work addresses scalable, privacy-conscious sparse linear regression under massive data by proposing the CESDAR algorithm, a distributed method that performs active-set selection and parameter updates using a communication-efficient surrogate likelihood. Each machine computes local gradients, Hessian-like terms, and coefficients, which are averaged on a master to update the active set, while a surrogate loss enables estimation without transmitting raw data; when $M=1$, CESDAR reduces to the ESDAR method. Theoretical results establish nonasymptotic $l_2$ and $l_\infty$ error bounds under SRC, mutual coherence, and sub-Gaussian noise, with finite iteration guarantees and oracle-equivalence under sufficient signal strength. Extensions ECESDAR and ACESDAR further reduce communication and adaptively select tuning parameters, respectively, and simulations plus real-data applications demonstrate substantial speedups with minimal loss in accuracy, making the approach practical for large-scale, high-dimensional problems.

Abstract

In this paper, we propose a communication-efficient penalized regression algorithm for high-dimensional sparse linear regression models with massive data. This approach incorporates an optimized distributed system communication algorithm, named CESDAR algorithm, based on the Enhanced Support Detection and Root finding algorithm. The CESDAR algorithm leverages data distributed across multiple machines to compute and update the active set and introduces the communication-efficient surrogate likelihood framework to approximate the optimal solution for the full sample on the active set, resulting in the avoidance of raw data transmission, which enhances privacy and data security, while significantly improving algorithm execution speed and substantially reducing communication costs. Notably, this approach achieves the same statistical accuracy as the global estimator. Furthermore, this paper explores an extended version of CESDAR and an adaptive version of CESDAR to enhance algorithmic speed and optimize parameter selection, respectively. Simulations and real data benchmarks experiments demonstrate the efficiency and accuracy of the CESDAR algorithm.

Communication-Efficient l_0 Penalized Least Square

TL;DR

This work addresses scalable, privacy-conscious sparse linear regression under massive data by proposing the CESDAR algorithm, a distributed method that performs active-set selection and parameter updates using a communication-efficient surrogate likelihood. Each machine computes local gradients, Hessian-like terms, and coefficients, which are averaged on a master to update the active set, while a surrogate loss enables estimation without transmitting raw data; when , CESDAR reduces to the ESDAR method. Theoretical results establish nonasymptotic and error bounds under SRC, mutual coherence, and sub-Gaussian noise, with finite iteration guarantees and oracle-equivalence under sufficient signal strength. Extensions ECESDAR and ACESDAR further reduce communication and adaptively select tuning parameters, respectively, and simulations plus real-data applications demonstrate substantial speedups with minimal loss in accuracy, making the approach practical for large-scale, high-dimensional problems.

Abstract

In this paper, we propose a communication-efficient penalized regression algorithm for high-dimensional sparse linear regression models with massive data. This approach incorporates an optimized distributed system communication algorithm, named CESDAR algorithm, based on the Enhanced Support Detection and Root finding algorithm. The CESDAR algorithm leverages data distributed across multiple machines to compute and update the active set and introduces the communication-efficient surrogate likelihood framework to approximate the optimal solution for the full sample on the active set, resulting in the avoidance of raw data transmission, which enhances privacy and data security, while significantly improving algorithm execution speed and substantially reducing communication costs. Notably, this approach achieves the same statistical accuracy as the global estimator. Furthermore, this paper explores an extended version of CESDAR and an adaptive version of CESDAR to enhance algorithmic speed and optimize parameter selection, respectively. Simulations and real data benchmarks experiments demonstrate the efficiency and accuracy of the CESDAR algorithm.

Paper Structure

This paper contains 14 sections, 5 theorems, 54 equations, 2 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. CESDAR algorithm
  3. Theoretical Properties
  4. $l_2$ error bounds
  5. $l_\infty$ error bounds
  6. Extension
  7. Tuning parameter selection
  8. Simulation Study
  9. Simulation Examples
  10. Simulation results and analysis
  11. Real Data Applications
  12. Avocado Price Data
  13. Human Resource Data
  14. Concluding Remarks

Key Result

Lemma 1

If $\hat{\bm{\beta}}$ is a global minimizer of (3), then $\hat{\bm{\beta}}$ satisfies where $l(\hat{\bm{\beta}}_{-i},\beta_i)$ represents the objective function of $\beta_i$ and $H_\lambda(\cdot)$ is defined by Conversely, if $\hat{\bm{\beta}}$, $\hat{\bm{d}}$ and $\hat{\bm{g}}$ satisfy (4), then $\hat{\bm{\beta}}$ is a local minimizer of (3).

Figures (2)

  • Figure 1: Simulation results for ESDAR, CESDAR and ECESDAR in Example \ref{['ex:3']}.
  • Figure 2: Simulation results for ESDAR, CESDAR and ECESDAR in Example \ref{['ex:4']}.

Theorems & Definitions (22)

  • Lemma 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Theorem 5
  • Remark 6
  • Theorem 7
  • Remark 8
  • Theorem 9
  • Remark 10
  • ...and 12 more