Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective

Yu-Bang Zheng, Xi-Le Zhao, Junhua Zeng, Chao Li, Qibin Zhao, Heng-Chao Li, Ting-Zhu Huang

TL;DR

The paper tackles the challenging TN-SS problem, where searching over TN topology and rank is NP-hard and traditional sampling-evaluation methods are computationally prohibitive. It introduces SVDinsTN, a regularized TN paradigm that inserts diagonal factors between adjacent cores in a fully-connected topology, enabling simultaneous core and diagonal optimization and making the diagonal sparsity a proxy for the TN structure. The optimization framework combines a least-squares data-fit term with $ rac{bc}{2}\sum_k rm{\mathcal{G}_k}_F^2$ and an $$-norm sparsity term on the diagonal factors, solved via a PAM_KL alternating scheme with ADMM for the diagonal factors; an initialization strategy based on truncated SVD provides a good starting point and theoretical convergence to a critical point is established. Empirically, SVDinsTN delivers about $100$ to $1000$ times acceleration over state-of-the-art TN-SS methods while maintaining comparable representation power and showing superior tensor completion performance on color videos, thus offering a practical route to efficient, structure-aware TN representations in high-order data analysis.

Abstract

Tensor network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a challenging NP-hard problem. Recent "sampling-evaluation"-based methods require sampling an extensive collection of structures and evaluating them one by one, resulting in prohibitively high computational costs. To address this issue, we propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling perspective, eliminating the repeated structure evaluations. To be specific, by inserting a diagonal factor for each edge of the fully-connected TN, SVDinsTN allows us to calculate TN cores and diagonal factors simultaneously, with the factor sparsity revealing a compact TN structure. In theory, we prove a convergence guarantee for the proposed method. Experimental results demonstrate that the proposed method achieves approximately 100 to 1000 times acceleration compared to the state-of-the-art TN-SS methods while maintaining a comparable level of representation ability.

SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective

TL;DR

The paper tackles the challenging TN-SS problem, where searching over TN topology and rank is NP-hard and traditional sampling-evaluation methods are computationally prohibitive. It introduces SVDinsTN, a regularized TN paradigm that inserts diagonal factors between adjacent cores in a fully-connected topology, enabling simultaneous core and diagonal optimization and making the diagonal sparsity a proxy for the TN structure. The optimization framework combines a least-squares data-fit term with and an -norm sparsity term on the diagonal factors, solved via a PAM_KL alternating scheme with ADMM for the diagonal factors; an initialization strategy based on truncated SVD provides a good starting point and theoretical convergence to a critical point is established. Empirically, SVDinsTN delivers about to times acceleration over state-of-the-art TN-SS methods while maintaining comparable representation power and showing superior tensor completion performance on color videos, thus offering a practical route to efficient, structure-aware TN representations in high-order data analysis.

Abstract

Tensor network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a challenging NP-hard problem. Recent "sampling-evaluation"-based methods require sampling an extensive collection of structures and evaluating them one by one, resulting in prohibitively high computational costs. To address this issue, we propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling perspective, eliminating the repeated structure evaluations. To be specific, by inserting a diagonal factor for each edge of the fully-connected TN, SVDinsTN allows us to calculate TN cores and diagonal factors simultaneously, with the factor sparsity revealing a compact TN structure. In theory, we prove a convergence guarantee for the proposed method. Experimental results demonstrate that the proposed method achieves approximately 100 to 1000 times acceleration compared to the state-of-the-art TN-SS methods while maintaining a comparable level of representation ability.
Paper Structure (13 sections, 2 theorems, 11 equations, 4 figures, 6 tables, 3 algorithms)

This paper contains 13 sections, 2 theorems, 11 equations, 4 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Notations and Preliminaries
  4. Tensor Network
  5. An Efficient Method for TN-SS
  6. SVDinsTN
  7. A Regularized Method for TN-SS
  8. Initialization Scheme
  9. Numerical Experiments
  10. Experiments for Validating Claim A
  11. Experiments for Validating Claim B
  12. Experiments for Validating Claim C
  13. Conclusion

Key Result

Theorem 1

The sequence generated by Algorithm AlgforSTNm, denoted by $\{\mathcal{G}^{(s)},\mathbf{S}^{(s)}\}_{s\in{\mathbb{N}}}$, converges to a critical point of the optimization problem (main_modelX2).

Figures (4)

  • Figure 1: (a) A graphical illustration of SVD. (b) A graphical illustration of SVD-inspired TN decomposition on a fifth-order tensor. (c) Comparison of the compression ratio ($\downarrow$) and run time ($\downarrow$) of different methods on a fifth-order light field image Knights, where the reconstruction error bound is set to 0.05, TRALS zhao2016tensor and FCTNALS zhengFCTN are methods with pre-defined topologies, and TNGreedy hashemizadeh2022adaptive, TNGA LiTopologySearch, TNLS LiPermutationSearch, and TNALE li2023alternating are TN-SS methods (please see more results in Table \ref{['FLpertab']}).
  • Figure 2: Comparison of the runtime in the first five iterations of SVDinsTN on light field data Truck when including and excluding the $\mathrm{shrink}$ operation in our initialization scheme.
  • Figure 3: Reconstructed images and residual images obtained by different methods on the 25th frame of News. Here the residual image is the average absolute difference between the reconstructed image and the ground truth over R, G, and B channels.
  • Figure 4: Relative change curves with respect to the iteration number on test color videos Bunny and Silent. Here the relative change is defined as $\|\mathcal{X}-\hat{\mathcal{X}}\|_F/\|\hat{\mathcal{X}}\|_F$, and $\mathcal{X}$ and $\hat{\mathcal{X}}$ are the results of the current iteration and its previous iteration.

Theorems & Definitions (5)

  • Definition 1: SVDinsTN
  • Remark 1: SVDinsTN & SVD
  • Remark 2: SVDinsTN & FCTN
  • Theorem 1: Convergence guarantee
  • Theorem 2