Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient algorithms for the Hadamard decomposition

Samuel Wertz, Arnaud Vandaele, Nicolas Gillis

TL;DR

The paper tackles efficient Hadamard decomposition by approximating X with the Hadamard product of low-rank factors, \tilde{X} = \bigl(W_1H_1\bigr) \circ \bigl(W_2H_2\bigr). It develops a block-coordinate descent solver where each factor update reduces to convex, column-separable Hadamard least-squares subproblems solved via hadLS, and it provides GD-based and exact solution options along with a detailed complexity analysis. It introduces several initialization schemes, notably an SVD-based method that couples a magnitude and sign decomposition, and augments the method with momentum-based acceleration. The framework is extended to more than two factors to increase expressiveness while controlling computational cost, and extensive experiments on synthetic and real data show favorable convergence and reconstruction accuracy relative to gradient-based baselines and, in sparse settings, to SVD with equivalent parameter budgets. Collectively, these contributions deliver a scalable, flexible approach for high-rank Hadamard approximations with practical impact on data compression and analysis tasks.

Abstract

The Hadamard decomposition is a powerful technique for data analysis and matrix compression, which decomposes a given matrix into the element-wise product of two or more low-rank matrices. In this paper, we develop an efficient algorithm to solve this problem, leveraging an alternating optimization approach that decomposes the global non-convex problem into a series of convex sub-problems. To improve performance, we explore advanced initialization strategies inspired by the singular value decomposition (SVD) and incorporate acceleration techniques by introducing momentum-based updates. Beyond optimizing the two-matrix case, we also extend the Hadamard decomposition framework to support more than two low-rank matrices, enabling approximations with higher effective ranks while preserving computational efficiency. Finally, we conduct extensive experiments to compare our method with the existing gradient descent-based approaches for the Hadamard decomposition and with traditional low-rank approximation techniques. The results highlight the effectiveness of our proposed method across diverse datasets.

Efficient algorithms for the Hadamard decomposition

TL;DR

The paper tackles efficient Hadamard decomposition by approximating X with the Hadamard product of low-rank factors, \tilde{X} = \bigl(W_1H_1\bigr) \circ \bigl(W_2H_2\bigr). It develops a block-coordinate descent solver where each factor update reduces to convex, column-separable Hadamard least-squares subproblems solved via hadLS, and it provides GD-based and exact solution options along with a detailed complexity analysis. It introduces several initialization schemes, notably an SVD-based method that couples a magnitude and sign decomposition, and augments the method with momentum-based acceleration. The framework is extended to more than two factors to increase expressiveness while controlling computational cost, and extensive experiments on synthetic and real data show favorable convergence and reconstruction accuracy relative to gradient-based baselines and, in sparse settings, to SVD with equivalent parameter budgets. Collectively, these contributions deliver a scalable, flexible approach for high-rank Hadamard approximations with practical impact on data compression and analysis tasks.

Abstract

The Hadamard decomposition is a powerful technique for data analysis and matrix compression, which decomposes a given matrix into the element-wise product of two or more low-rank matrices. In this paper, we develop an efficient algorithm to solve this problem, leveraging an alternating optimization approach that decomposes the global non-convex problem into a series of convex sub-problems. To improve performance, we explore advanced initialization strategies inspired by the singular value decomposition (SVD) and incorporate acceleration techniques by introducing momentum-based updates. Beyond optimizing the two-matrix case, we also extend the Hadamard decomposition framework to support more than two low-rank matrices, enabling approximations with higher effective ranks while preserving computational efficiency. Finally, we conduct extensive experiments to compare our method with the existing gradient descent-based approaches for the Hadamard decomposition and with traditional low-rank approximation techniques. The results highlight the effectiveness of our proposed method across diverse datasets.

Paper Structure

This paper contains 23 sections, 3 theorems, 15 equations, 6 figures, 4 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. BCD for the Hadamard Decomposition
  3. Solving hadLS \ref{['eq:obj']}
  4. Gradient descent (GD) based iterative methods
  5. Exact solution
  6. Complexity analysis
  7. Initializations
  8. Xavier Initialization
  9. SVD-Based Initialization
  10. K-Means Initialization
  11. Optimal scaling
  12. Acceleration by Adding Momentum
  13. More than two matrices in the decomposition
  14. Numerical experiments
  15. Comparison with the previous method from b2
  16. ...and 8 more sections

Key Result

Theorem 1

Let $R = \sum_{i=1}^p r_i$ be the budgetWe call this the budget because two decompositions with the same budget have the same number of parameters, in the factors $(W_i, H_i)$'s, namely $\sum_{i=1}^p r_i(m+n) = R (m+n)$. of an Hadamard decomposition with $p$ matrices of rank $r_i$. Such a decomposit In such decompositions, $r_i \in \{2,3\}$ for all $i$.

Figures (6)

  • Figure 1: Evolution of the normalized average error for the five initializations for BCD with exact resolution.
  • Figure 2: Impact of extrapolation on BCD with exact resolution.
  • Figure 3: Performance comparison of our methods and AGD b2 on synthetic datasets.
  • Figure 4: Performance comparison of our method and AGD b2 on synthetic datasets.
  • Figure 5: Performance comparison of our method and AGD b2 on real datasets.
  • ...and 1 more figures

Theorems & Definitions (7)

  • Theorem 1
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • proof : Proof of Theorem \ref{['th:identity']}