Tensor Network Structure Search with Program Synthesis

TL;DR

This work addresses tensor network structure search (TN-SS) by reframing it as transformation program synthesis. It introduces output-directed splits to prune the search space, and a constraint-based ranking pipeline that avoids performing full tensor decompositions for most candidates. The method decomposes the TN-SS workflow into four phases: singular value precomputation, sketch-program generation, cost estimation via integer programming, and final decomposition/rounding, achieving up to 10x faster search and 1.5x–3x better compression than state-of-the-art on large tensors, with demonstrated generalization to unseen data. The approach offers scalable, principled TN-SS that is applicable to machine learning, scientific computing, and related domains where efficient data compression is essential.

Abstract

Tensor networks provide a powerful framework for compressing multi-dimensional data. The optimal tensor network structure for a given data tensor depends on both data characteristics and specific optimality criteria, making tensor network structure search a difficult problem. Existing solutions typically rely on sampling and compressing numerous candidate structures; these procedures are computationally expensive and therefore limiting for practical applications. We address this challenge by viewing tensor network structure search as a program synthesis problem and introducing an efficient constraint-based assessment method that avoids costly tensor decomposition. Specifically, we establish a correspondence between transformation programs and network structures. We also design a novel operation named output-directed splits to reduce the search space without hindering expressiveness. We then propose a synthesis algorithm to identify promising network candidates through constraint solving, and avoid tensor decomposition for all but the most promising candidates. Experimental results show that our approach improves search speed by up to $10\times$ and achieves compression ratios $1.5\times$ to $3\times$ better than state-of-the-art. Notably, our approach scales to larger tensors that are unattainable by prior work. Furthermore, the discovered topologies generalize well to similar data, yielding compression ratios up to $ 2.4\times$ better than a generic structure while the runtime remains around $110$ seconds.

Figures (8)

• Figure 1: The overall workflow. We use green for inputs, purple for preprocess, orange for sketch enumeration, pink for score computation, and blue for decomposition, rounding, and outputs.
• Figure 2: The structure search algorithm
• Figure 3: The suboptimal resulting structure of input-directed splits.
• Figure 5: Comparison of compression ratio vs time on real datasets. The datasets from left to right are light field data ($40 \times 60 \times 3 \times 9 \times 9$), BigEarthNet ($30 \times 12 \times 120 \times 120$), BigEarthNet ($5 \times 20 \times 30 \times 12 \times 120 \times 120$), and PDEBench ($10 \times 5 \times 21 \times 64 \times 64 \times 64$). The two rows corresponds to error bounds of $0.1$ and $0.01$ respectively.
• Figure 6: Comparison of compression ratios for random generated data.

Theorems & Definitions (24)

• Definition 3.1: Tensor, Tensor Size
• Definition 3.2: Matricization
• Definition 3.3: Tensor Network
• Definition 3.4: Tensor Network Structure Search
• Definition 3.5: Program
• Definition 3.6: Sketch Program, Rank Assignment
• Theorem 3.7: Upper Bound of Singular Values
• Lemma A.1
• proof
• Theorem A.2: Completeness of Output-Directed Splits