Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable High-Dimensional Multivariate Linear Regression for Feature-Distributed Data

Shuo-Chieh Huang, Ruey S. Tsay

TL;DR

The proposed two-stage relaxed greedy algorithm (TSRGA) for applying multivariate linear regression to feature-distributed data is applied in a financial application that leverages unstructured data from the 10-K reports, demonstrating its usefulness in applications with many dense large-dimensional matrices.

Abstract

Feature-distributed data, referred to data partitioned by features and stored across multiple computing nodes, are increasingly common in applications with a large number of features. This paper proposes a two-stage relaxed greedy algorithm (TSRGA) for applying multivariate linear regression to such data. The main advantage of TSRGA is that its communication complexity does not depend on the feature dimension, making it highly scalable to very large data sets. In addition, for multivariate response variables, TSRGA can be used to yield low-rank coefficient estimates. The fast convergence of TSRGA is validated by simulation experiments. Finally, we apply the proposed TSRGA in a financial application that leverages unstructured data from the 10-K reports, demonstrating its usefulness in applications with many dense large-dimensional matrices.

Scalable High-Dimensional Multivariate Linear Regression for Feature-Distributed Data

TL;DR

The proposed two-stage relaxed greedy algorithm (TSRGA) for applying multivariate linear regression to feature-distributed data is applied in a financial application that leverages unstructured data from the 10-K reports, demonstrating its usefulness in applications with many dense large-dimensional matrices.

Abstract

Feature-distributed data, referred to data partitioned by features and stored across multiple computing nodes, are increasingly common in applications with a large number of features. This paper proposes a two-stage relaxed greedy algorithm (TSRGA) for applying multivariate linear regression to such data. The main advantage of TSRGA is that its communication complexity does not depend on the feature dimension, making it highly scalable to very large data sets. In addition, for multivariate response variables, TSRGA can be used to yield low-rank coefficient estimates. The fast convergence of TSRGA is validated by simulation experiments. Finally, we apply the proposed TSRGA in a financial application that leverages unstructured data from the 10-K reports, demonstrating its usefulness in applications with many dense large-dimensional matrices.
Paper Structure (22 sections, 11 theorems, 156 equations, 8 figures, 4 tables, 2 algorithms)

This paper contains 22 sections, 11 theorems, 156 equations, 8 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Distributed framework and two-stage relaxed greedy algorithm
  3. Model and distributed framework
  4. First-stage relaxed greedy algorithm and a just-in-time stopping criterion
  5. Second-stage relaxed greedy algorithm
  6. Related algorithms
  7. Communication complexity of TSRGA
  8. Assumptions
  9. Main results
  10. Simulation experiments
  11. Statistical performance of TSRGA
  12. Large-scale performance of TSRGA
  13. Empirical application
  14. Financial data and 10-K reports
  15. Results
  16. ...and 7 more sections

Key Result

Theorem 1

Assume (C1)-(C4) hold. Suppose there exists an $M_o<\infty$ such that $M_o^{-1} \leq (nd_{n})^{-1}\Vert \mathbf{E} \Vert_{F}^{2} \leq M_o$ with probability tending to one. Write $\hat{\mathbf{G}}^{(k)} = \sum_{j=1}^{p_{n}}\mathbf{X}_{j}\hat{\mathbf{B}}_{j}^{(k)}$, $k=1,2,\ldots,K_{n}$, for the itera

Figures (8)

  • Figure 1: Logarithm of parameter estimation errors of various methods under Specification 1, where $n$ is the sample size and $p_n$ is the dimension of predictors. The results are averages of 100 simulations.
  • Figure 2: Parameter estimation errors of various estimation methods under Specification 2, where $n$ is the sample size and $p_n$ is the number of predictors. The results are averages of 100 simulations.
  • Figure 3: Logarithm of the average parameter estimation errors at each iteration of TSRGA, plotted against the average time elapsed at the end of each iteration. Various number of processes are employed for feature-distributed implementation. 10 simulations are used.
  • Figure 4: Logarithm of the estimation errors of TSRGA (running with 16 processes) and the oracle least squares. The oracle least squares method is performed by applying the second-stage RGA with exactly the relevant predictors and no rank constraints. 10 simulations are used.
  • Figure 5: Logarithm of parameter estimation errors of various methods against the elapsed time under Specification 1, where $n$ is the sample size and $p_n$ is the dimension of predictors. The results are based on 100 simulations.
  • ...and 3 more figures

Theorems & Definitions (17)

  • Theorem 1
  • Lemma 2
  • Theorem 3
  • Corollary 4
  • Example 1: High-dimensional sparse linear regression
  • Example 2: Multi-task linear regression with common relevant predictors
  • Example 3: Integrative multi-view regression
  • Corollary 5
  • Lemma 6
  • Theorem 7
  • ...and 7 more