Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks

Tahseen Rabbani, Jiahao Su, Xiaoyu Liu, David Chan, Geoffrey Sangston, Furong Huang

TL;DR

The paper tackles the efficiency bottlenecks of tensorized convolutional networks by introducing conv_einsum, an einsum-like framework that represents and optimizes the evaluation of multilinear operations including convolutions. It extends the netcon-based optimal sequencing to handle convolutions, enabling FLOP-minimal execution paths for a wide range of tensor decompositions (CP, Tucker, TT, TR, BT, HT, and reshaped variants) and network designs (e.g., interleaved group and depthwise convolutions). Through theory and extensive experiments, the authors show significant improvements in computational and memory efficiency across tasks such as ASR, video classification, and image classification, with an open-source implementation for training tensorized networks. The work provides a practical, scalable path toward deploying compact, expressive neural nets on resource-constrained devices. Future directions include integrating advanced accelerators and stabilization techniques to further boost performance and stability of tensorized architectures.

Abstract

Modern ConvNets continue to achieve state-of-the-art results over a vast array of vision and image classification tasks, but at the cost of increasing parameters. One strategy for compactifying a network without sacrificing much expressive power is to reshape it into a tensorial neural network (TNN), which is a higher-order tensorization of its layers, followed by a factorization, such as a CP-decomposition, which strips a weight down to its critical basis components. Passes through TNNs can be represented as sequences of multilinear operations (MLOs), where the evaluation path can greatly affect the number of floating point operations (FLOPs) incurred. While functions such as the popular einsum can evaluate simple MLOs such as contractions, existing implementations cannot process multi-way convolutions, resulting in scant assessments of how optimal evaluation paths through tensorized convolutional layers can improve training speed. In this paper, we develop a unifying framework for representing tensorial convolution layers as einsum-like strings and a meta-algorithm conv_einsum which is able to evaluate these strings in a FLOPs-minimizing manner. Comprehensive experiments, using our open-source implementation, over a wide range of models, tensor decompositions, and diverse tasks, demonstrate that conv_einsum significantly increases both computational and memory-efficiency of convolutional TNNs.

conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks

TL;DR

The paper tackles the efficiency bottlenecks of tensorized convolutional networks by introducing conv_einsum, an einsum-like framework that represents and optimizes the evaluation of multilinear operations including convolutions. It extends the netcon-based optimal sequencing to handle convolutions, enabling FLOP-minimal execution paths for a wide range of tensor decompositions (CP, Tucker, TT, TR, BT, HT, and reshaped variants) and network designs (e.g., interleaved group and depthwise convolutions). Through theory and extensive experiments, the authors show significant improvements in computational and memory efficiency across tasks such as ASR, video classification, and image classification, with an open-source implementation for training tensorized networks. The work provides a practical, scalable path toward deploying compact, expressive neural nets on resource-constrained devices. Future directions include integrating advanced accelerators and stabilization techniques to further boost performance and stability of tensorized architectures.

Abstract

Modern ConvNets continue to achieve state-of-the-art results over a vast array of vision and image classification tasks, but at the cost of increasing parameters. One strategy for compactifying a network without sacrificing much expressive power is to reshape it into a tensorial neural network (TNN), which is a higher-order tensorization of its layers, followed by a factorization, such as a CP-decomposition, which strips a weight down to its critical basis components. Passes through TNNs can be represented as sequences of multilinear operations (MLOs), where the evaluation path can greatly affect the number of floating point operations (FLOPs) incurred. While functions such as the popular einsum can evaluate simple MLOs such as contractions, existing implementations cannot process multi-way convolutions, resulting in scant assessments of how optimal evaluation paths through tensorized convolutional layers can improve training speed. In this paper, we develop a unifying framework for representing tensorial convolution layers as einsum-like strings and a meta-algorithm conv_einsum which is able to evaluate these strings in a FLOPs-minimizing manner. Comprehensive experiments, using our open-source implementation, over a wide range of models, tensor decompositions, and diverse tasks, demonstrate that conv_einsum significantly increases both computational and memory-efficiency of convolutional TNNs.
Paper Structure (26 sections, 4 theorems, 13 equations, 11 figures, 7 tables)

This paper contains 26 sections, 4 theorems, 13 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Tensor Operations and einsum
  3. Representations of Multilinear Operations
  4. From einsum to conv_einsum
  5. Compact Neural Networks via conv_einsum
  6. Algorithms
  7. Atomic Operations
  8. Optimal Sequencer
  9. Checkpointing
  10. Related works
  11. Experiments
  12. Runtime and FLOPs results
  13. Memory Results
  14. Conclusions and Discussions
  15. Supplementary Material for Tensor Operations and einsum
  16. ...and 11 more sections

Key Result

Theorem 1

Let $\bm{\mathcal{X}}\in\mathbb{R}^{B \times S\times H' \times W'}$ be the input to a reshaped CP (RCP) convolutional kernel $\bm{\mathcal{\overline{W}}} \in \mathbb{R}^{T_1 \cdots \times T_M \times S_1 \cdots \times S_M \times H \times W}$ such that $T = \prod_{m = 1}^{M} T_m$, $S = \prod_{m = 1}^{ has a pairwise evaluation path which costs less FLOPs than the naive left-to-right evaluation, whe

Figures (11)

  • Figure 1: Tensor sequence generation. We analyze a sequence over a collection of tensors $\bm{\mathcal{A}}, \bm{\mathcal{B}}, \bm{\mathcal{C}}$, $\bm{\mathcal{D}}$, involving contraction, convolution, and batch product (Python). We store the optimal sequence in a string array path_info.
  • Figure 2: An optimal sequence of paths. Visualization (via opt-einsum) of the optimal sequence of paths for the conv_einsum string submitted in \ref{['fig:cvoutput-a']}, with our modified support for convolution symbols. We are also presented with information about the naive left-to-right cost vs the cost of the suggested path.
  • Figure 4: Optimal sequencer example. conv_einsum deploys the optimal sequencer to analyze the path tree of an abstract tensor sequence $\bm{\mathcal{A}} \circ_1 \bm{\mathcal{B}} \circ_2 \bm{\mathcal{C}} \circ_3 \bm{\mathcal{D}} \circ_4 \bm{\mathcal{E}}$, where $\circ_i$ for $1\leq i \leq 4$ is any collection of multi-linear operations, including convolutions, batch products, contractions, and outer products. The green path indicates the optimal path and the orange path indicates a path which satisfies a user-specified cost cap $c$ at each node.
  • Figure 5: ASR Train
  • Figure 6: ASR Test
  • ...and 6 more figures

Theorems & Definitions (6)

  • Theorem 1: CP reduction
  • Theorem 2: Tucker reduction
  • Theorem : CP reduction
  • proof
  • Theorem : Tucker reduction
  • proof