Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tensor Evolution: A Framework for Fast Evaluation of Tensor Computations using Recurrences

Javed Absar, Samarth Narang, Muthu Baskaran

TL;DR

Tensor Evolution (TeV) generalizes the Scalar Evolution framework to tensors, enabling analysis of how tensor values evolve inside loops via chains of recurrences and a set of algebraic rewrite lemmas. The approach defines basic and chained tensor evolutions, provides formal lemmas for reshaping, slicing, concatenation, and combining TeV expressions, and demonstrates closed-form transformations in concrete code. The work relates TeV to existing representations like Tensor Comprehensions and Halide, while highlighting TeV's unique focus on reducibility and algebraic simplification for loop-carried tensor computations. If extended, TeV could power loop-strength reduction and exit-value optimization in ML/HPC compilers and DSLs, improving performance in practical tensor-heavy workloads.

Abstract

This paper introduces a new mathematical framework for analysis and optimization of tensor expressions within an enclosing loop. Tensors are multi-dimensional arrays of values. They are common in high performance computing (HPC) and machine learning domains. Our framework extends Scalar Evolution - an important optimization pass implemented in both LLVM and GCC - to tensors. Scalar Evolution (SCEV) relies on the theory of `Chain of Recurrences' for its mathematical underpinnings. We use the same theory for Tensor Evolution (TeV). While some concepts from SCEV map easily to TeV -- e.g. element-wise operations; tensors introduce new operations such as concatenation, slicing, broadcast, reduction, and reshape which have no equivalent in scalars and SCEV. Not all computations are amenable to TeV analysis but it can play a part in the optimization and analysis parts of ML and HPC compilers. Also, for many mathematical/compiler ideas, applications may go beyond what was initially envisioned, once others build on it and take it further. We hope for a similar trajectory for the tensor-evolution concept.

Tensor Evolution: A Framework for Fast Evaluation of Tensor Computations using Recurrences

TL;DR

Tensor Evolution (TeV) generalizes the Scalar Evolution framework to tensors, enabling analysis of how tensor values evolve inside loops via chains of recurrences and a set of algebraic rewrite lemmas. The approach defines basic and chained tensor evolutions, provides formal lemmas for reshaping, slicing, concatenation, and combining TeV expressions, and demonstrates closed-form transformations in concrete code. The work relates TeV to existing representations like Tensor Comprehensions and Halide, while highlighting TeV's unique focus on reducibility and algebraic simplification for loop-carried tensor computations. If extended, TeV could power loop-strength reduction and exit-value optimization in ML/HPC compilers and DSLs, improving performance in practical tensor-heavy workloads.

Abstract

This paper introduces a new mathematical framework for analysis and optimization of tensor expressions within an enclosing loop. Tensors are multi-dimensional arrays of values. They are common in high performance computing (HPC) and machine learning domains. Our framework extends Scalar Evolution - an important optimization pass implemented in both LLVM and GCC - to tensors. Scalar Evolution (SCEV) relies on the theory of `Chain of Recurrences' for its mathematical underpinnings. We use the same theory for Tensor Evolution (TeV). While some concepts from SCEV map easily to TeV -- e.g. element-wise operations; tensors introduce new operations such as concatenation, slicing, broadcast, reduction, and reshape which have no equivalent in scalars and SCEV. Not all computations are amenable to TeV analysis but it can play a part in the optimization and analysis parts of ML and HPC compilers. Also, for many mathematical/compiler ideas, applications may go beyond what was initially envisioned, once others build on it and take it further. We hope for a similar trajectory for the tensor-evolution concept.

Paper Structure

This paper contains 10 sections, 13 theorems, 21 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Chain of Recurrences and Scalar Evolution
  3. Basic Recurrence Lemmas
  4. Tensor Evolution
  5. Code Example
  6. Basic Formulation
  7. Tensor Evolution Lemmas
  8. Application of Tensor Evolution
  9. Background Art and Related Work
  10. Conclusion

Key Result

Lemma 1

For a constant $c$ added to a basic recurrence:

Figures (3)

  • Figure 1: Basic Tensor Evolution
  • Figure 2: Adding two TeV expressions
  • Figure 3: Calculate tensor evolution expressions of given code.

Theorems & Definitions (13)

  • Lemma 1: Addition of a Constant
  • Lemma 2: Multiplication by a Constant
  • Lemma 3: Exponentiation by a Constant
  • Lemma 4: Addition of Two Recurrences
  • Lemma 5: Multiplication of Two Recurrences
  • Lemma 6: Chain of Recurrences
  • Lemma 7
  • Lemma 8
  • Lemma 9
  • Lemma 10
  • ...and 3 more