Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reweighted Solutions for Weighted Low Rank Approximation

David P. Woodruff, Taisuke Yasuda

TL;DR

This paper tackles weighted low rank approximation (WLRA), an NP-hard problem, by introducing a simple reweighting relaxation that uses the weight matrix $\mathbf{W}$ to reweight a low-rank solution. Under the natural assumption that $\mathrm{rank}(\mathbf{W})\le r$, a rank-$rk$ approximation of $\mathbf{W}\circ\mathbf{A}$ combined with an entrywise division by $\mathbf{W}$ yields provable approximation guarantees, with near-optimal storage and efficiency via randomized SVD. The authors establish a first relative-error guarantee for weighted feature selection via column subset selection and show nearly tight communication complexity bounds that depend on $r$, the rank of $\mathbf{W}$. They also demonstrate practical performance in model compression tasks, where weight matrices exhibit low-rank structure, and provide experiments on synthetic data that corroborate the theoretical findings. Overall, the work offers a simple yet powerful framework that unifies theory and practice for WLRA, with implications for distributed computing and parameter-efficient weight representations in large models.

Abstract

Weighted low rank approximation (WLRA) is an important yet computationally challenging primitive with applications ranging from statistical analysis, model compression, and signal processing. To cope with the NP-hardness of this problem, prior work considers heuristics, bicriteria, or fixed parameter tractable algorithms to solve this problem. In this work, we introduce a new relaxed solution to WLRA which outputs a matrix that is not necessarily low rank, but can be stored using very few parameters and gives provable approximation guarantees when the weight matrix has low rank. Our central idea is to use the weight matrix itself to reweight a low rank solution, which gives an extremely simple algorithm with remarkable empirical performance in applications to model compression and on synthetic datasets. Our algorithm also gives nearly optimal communication complexity bounds for a natural distributed problem associated with this problem, for which we show matching communication lower bounds. Together, our communication complexity bounds show that the rank of the weight matrix provably parameterizes the communication complexity of WLRA. We also obtain the first relative error guarantees for feature selection with a weighted objective.

Reweighted Solutions for Weighted Low Rank Approximation

TL;DR

This paper tackles weighted low rank approximation (WLRA), an NP-hard problem, by introducing a simple reweighting relaxation that uses the weight matrix to reweight a low-rank solution. Under the natural assumption that , a rank- approximation of combined with an entrywise division by yields provable approximation guarantees, with near-optimal storage and efficiency via randomized SVD. The authors establish a first relative-error guarantee for weighted feature selection via column subset selection and show nearly tight communication complexity bounds that depend on , the rank of . They also demonstrate practical performance in model compression tasks, where weight matrices exhibit low-rank structure, and provide experiments on synthetic data that corroborate the theoretical findings. Overall, the work offers a simple yet powerful framework that unifies theory and practice for WLRA, with implications for distributed computing and parameter-efficient weight representations in large models.

Abstract

Weighted low rank approximation (WLRA) is an important yet computationally challenging primitive with applications ranging from statistical analysis, model compression, and signal processing. To cope with the NP-hardness of this problem, prior work considers heuristics, bicriteria, or fixed parameter tractable algorithms to solve this problem. In this work, we introduce a new relaxed solution to WLRA which outputs a matrix that is not necessarily low rank, but can be stored using very few parameters and gives provable approximation guarantees when the weight matrix has low rank. Our central idea is to use the weight matrix itself to reweight a low rank solution, which gives an extremely simple algorithm with remarkable empirical performance in applications to model compression and on synthetic datasets. Our algorithm also gives nearly optimal communication complexity bounds for a natural distributed problem associated with this problem, for which we show matching communication lower bounds. Together, our communication complexity bounds show that the rank of the weight matrix provably parameterizes the communication complexity of WLRA. We also obtain the first relative error guarantees for feature selection with a weighted objective.
Paper Structure (17 sections, 12 theorems, 25 equations, 6 figures, 5 tables, 1 algorithm)

This paper contains 17 sections, 12 theorems, 25 equations, 6 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our results
  3. Column subset selection for weighted low rank approximation
  4. Nearly optimal communication complexity bounds
  5. Experiments
  6. Related work
  7. Approximation algorithms
  8. Matrices with structured entrywise inverses
  9. Communication complexity bounds
  10. Experiments
  11. The low rank weight matrix assumption in practice
  12. Approximation quality and running time
  13. Experiments on synthetic datasets
  14. Conclusion
  15. Missing proofs from Section \ref{['sec:comm-comp']}
  16. ...and 2 more sections

Key Result

Theorem 1.2

Let $\mathbf{W}\in\mathbb R^{n\times d}$ be a non-negative weight matrix with rank $r$. Let $\mathbf{A}\in\mathbb R^{n\times d}$ and let $k\in\mathbb N$. Suppose that $\tilde{\mathbf{A}}_\mathbf{W}\in\mathbb R^{n\times d}$ satisfies and let $\tilde{\mathbf{A}}\coloneqq \mathbf{W}^{\circ-1}\circ\tilde{\mathbf{A}}_\mathbf{W}$, where $\mathbf{W}^{\circ-1}\in\mathbb R^{n\times d}$ denotes the entrywi

Figures (6)

  • Figure 1: Low rank structure of Fisher weight matrices
  • Figure 2: Fisher-weighted low rank approximation loss of weighted low rank approximation algorithms for model compression of four datasets. Results are averaged over $5$ trials.
  • Figure 3: Running time of weighted low rank approximation algorithms for model compression of four datasets. Results are averaged over $5$ trials.
  • Figure 4: Improving the svd_w solution with em iterations for a rank $20$ approximation.
  • Figure 5: Loss and running time of WLRA on a synthetic dataset based on a mixture of Gaussians. Results are averaged over $5$ trials.
  • ...and 1 more figures

Theorems & Definitions (25)

  • Definition 1.1: Approximate weighted low rank approximation
  • Theorem 1.2
  • Lemma 1.2
  • Corollary 1.2
  • Corollary 1.3: Column subset selection for weighted low rank approximation
  • proof
  • Definition 1.4: WLRA: communication game
  • Theorem 1.5
  • Lemma 2.1
  • proof
  • ...and 15 more