Encoding architecture algebra

TL;DR

An algebraic approach to constructing input-encoding architectures that properly account for the data's structure is introduced, providing a step toward achieving more typeful machine learning.

Abstract

Despite the wide variety of input types in machine learning, this diversity is often not fully reflected in their representations or model architectures, leading to inefficiencies throughout a model's lifecycle. This paper introduces an algebraic approach to constructing input-encoding architectures that properly account for the data's structure, providing a step toward achieving more typeful machine learning.

Figures (6)

• Figure 1: The tensor network diagram for the tensor MFL \ref{['eq:augmentedTensorFlattening']}. Shapes are tensors, lines represent indices, and connected lines are contracted indices.
• Figure 2: The tensor network diagram for the product type MFL \ref{['eq:encodingProductMultilinear']}. Shapes are tensors, lines represent indices, and connected lines are contracted indices.
• Figure 3: The type tree of the flattening of the product type \ref{['eq:example1']}. We used the type synonyms: $\texttt{Option}[\, \texttt{Scal} \,] = \texttt{Sum}[\, \texttt{Unit},\, \texttt{Scal} \,]$ and $\texttt{Bool} = \texttt{Sum}[\, \texttt{Unit},\, \texttt{Unit} \,]$.
• Figure 4: A simplified type tree of the flattening of \ref{['eq:example1']}.
• Figure 5: The tensor network diagram of the flattening architecture for list types \ref{['eq:flatteningList']}. Shapes are tensors, lines represent indices, and connected lines are contracted indices.