TenExp: Mixture-of-Experts-Based Tensor Decomposition Structure Search Framework

TL;DR

A mixture-of-experts-based tensor decomposition structure search framework (termed as TenExp), which allows us to dynamically select and activate suitable tensor decompositions in an unsupervised fashion and provides the approximation error bound of TenExp, which reveals the approximation capability of TenExp.

Abstract

Recently, tensor decompositions continue to emerge and receive increasing attention. Selecting a suitable tensor decomposition to exactly capture the low-rank structures behind the data is at the heart of the tensor decomposition field, which remains a challenging and relatively under-explored problem. Current tensor decomposition structure search methods are still confined by a fixed factor-interaction family (e.g., tensor contraction) and cannot deliver the mixture of decompositions. To address this problem, we elaborately design a mixture-of-experts-based tensor decomposition structure search framework (termed as TenExp), which allows us to dynamically select and activate suitable tensor decompositions in an unsupervised fashion. This framework enjoys two unique advantages over the state-of-the-art tensor decomposition structure search methods. Firstly, TenExp can provide a suitable single decomposition beyond a fixed factor-interaction family. Secondly, TenExp can deliver a suitable mixture of decompositions beyond a single decomposition. Theoretically, we also provide the approximation error bound of TenExp, which reveals the approximation capability of TenExp. Extensive experiments on both synthetic and realistic datasets demonstrate the superiority of the proposed TenExp compared to the state-of-the-art tensor decomposition-based methods.

Figures (6)

• Figure 1: Diagram of our TenExp. First, we perform the cross-factor-interaction energy-based rank estimation scheme for the input data. Then, the estimated ranks are fed into the cross-factor-interaction candidate search set. Finally, the top-$k$ gating mechanism selects the candidate tensor decompositions, leading to either a single decomposition ($k=1$) or a mixture of decompositions ($k>1$).
• Figure 2: Comparison between candidate tensor decompositions and our proposed TenExp in terms of relative error for fitting synthetic data. "Mixture of Decompositions" denotes a uniform mixture strategy for Tucker decomposition, FCTN decomposition, and TF. For simplicity, we uniformly set all ranks of Tucker decomposition, FCTN decomposition, and TF as the same values.
• Figure 3: The completion results of MSIs by different methods on Flowers, Beers, Balloons, and Feathers with SR = 0.1.
• Figure 4: The completion results of realistic color videos by different methods on News, Claire, Grandma, and Akiyo with SR = 0.1.
• Figure 5: The completion results of light field data by different methods on Greek, Museum, Medieval2, and Vinyl with SR = 0.1.

Theorems & Definitions (18)

• Definition 1: Generalized Unfolding fctn
• Definition 2: Outer Product sr
• Definition 3: Mode-$n$ Product sr
• Definition 4: Tensor Contraction fctn
• Definition 5: T-Product 34234014
• Definition 6: CP Decomposition cp
• Definition 7: Tucker Decomposition tucker
• Definition 8: TT Decomposition doi:10.1137/090752286ZHENG2025107808
• Definition 9: TR Decomposition Zhao2016TensorRDZHENG2025107808
• Definition 10: FCTN Decomposition fctnZHENG2025107808