The Fascinating World of 2 $\times$ 2 $\times$ 2 Tensors: Its Geometry and Optimization Challenges

TL;DR

This work analyzes the geometry and optimization of $2 \times 2 \times 2$ tensors, highlighting key deviations from matrix intuition through the lens of algebraic geometry and group actions. It develops a canonical-form framework via $\mathrm{GL}(2)^{\times 3}$ orbits, with the hyperdeterminant and multilinear rank jointly characterizing rank and typical ranks, revealing two typical ranks ($2$ and $3$) and a non-hierarchical closure structure. The paper demonstrates ill-posedness in low-rank tensor approximation: no best rank-2 approximation exists for rank-3 tensors in this size, a phenomenon tied to singularities on the hyperdeterminant zero surface and border-rank behavior. It also provides constructive algorithms for canonical form conversion and visualizations to build intuition, with extensive discussion on implications for larger tensors and numerical optimization.

Abstract

This educational article highlights the geometric and algebraic complexities that distinguish tensors from matrices, to supplement coverage in advanced courses on linear algebra, matrix analysis, and tensor decompositions. Using the case of real-valued 2 $\times$ 2 $\times$ 2 tensors, we show how tensors violate many well-known properties of matrices: (1) The rank of a matrix is bounded by its smallest dimension, but a 2 $\times$ 2 $\times$ 2 tensor can be rank 3. (2) Matrices have a single typical rank, but the rank of a generic 2 $\times$ 2 $\times$ 2 tensor can be 2 or 3 - it has two typical ranks. (3) Any limit point of a sequence of matrices of rank $r$ is at most rank $r$, but a limit point of a sequence of 2 $\times$ 2 $\times$ 2 tensors of rank 2 can be rank 3 (a higher rank). (4) Matrices always have a best rank-$r$ approximation, but no rank-3 tensor of size 2 $\times$ 2 $\times$ 2 has a best rank-2 approximation. We unify the analysis of the matrix and tensor cases using tools from algebraic geometry and optimization, providing derivations of these surprising facts. To build intuition for the geometry of rank-constrained sets, students and educators can explore the geometry of matrix and tensor ranks via our interactive visualization tool.

Figures (10)

• Figure 1: Matrix and tensor rank decompositions.
• Figure 1: Orbit (blue circle) of $\mathbf{x} = 10 \in \mathbb{R}^2$ (orange dot) under the action of $\operatorname{O}()(2)$
• Figure 1: Hasse diagram for a partitioning of $[0,2] \times [0,2] \subset \mathbb{R}^2$ partially ordered by inclusion of closures.
• Figure 1: Tangent and normal cones with respect to $\Omega \equiv \set{\mathbf{x} \in \mathbb{R}^2 : x_1^2+x_1^3-x_2^2 \geq 0}$.
• Figure 2: A spherically-clipped 3-dimensional projection of the 8-dimensional space of $2 \times 2 \times 2$ tensors showing the 7-dimensional zero set of the hyperdeterminant.

Theorems & Definitions (29)

• Example 3.2
• Proposition 3.4: Matrix orbits
• Example 3.7: TTM
• Theorem 3.9: Tensor orbits de_silva_tensor_2008
• Proof 1
• Example 3.12: Fast algorithm for multiplying complex numbers
• Proposition 4.1: consequence of basu_algorithms_2006
• Proposition 4.2: Dimension of space of rank-$r$ matrices
• Proof 2
• Theorem 4.3