The Real Tropical Geometry of Neural Networks

TL;DR

This work connects neural networks with piecewise linear activations to real tropical geometry by viewing network functions as tropical rational functions $g\oslash h$. It introduces two complementary parameter-space decompositions: (i) semialgebraic stratifications by fixed decision-boundary combinatorics and (ii) a classification fan built from activation patterns, both reflecting the tropical structure of CPWL networks. The authors prove a semialgebraic embedding for fixed architectures, describe the activation polytope and its normal fan as a geometric framework for CPWL classifiers, and show that decision boundaries correspond to tropical hypersurfaces dual to Newton polytope subdivisions. They establish that linear classifiers yield connected 0/1-loss sublevel sets, while general ReLU networks can produce disconnected level sets, and they formulate a tropical-oriented matroid-like axiomatization to capture activation patterns, offering a rigorous discrete-geometric lens on neural-network decision regions with potential implications for VC-dimension and optimization landscapes.

Abstract

We consider a binary classifier defined as the sign of a tropical rational function, that is, as the difference of two convex piecewise linear functions. The parameter space of ReLU neural networks is contained as a semialgebraic set inside the parameter space of tropical rational functions. We initiate the study of two different subdivisions of this parameter space: a subdivision into semialgebraic sets, on which the combinatorial type of the decision boundary is fixed, and a subdivision into a polyhedral fan, capturing the combinatorics of the partitions of the dataset. The sublevel sets of the 0/1-loss function arise as subfans of this classification fan, and we show that the level-sets are not necessarily connected. We describe the classification fan i) geometrically, as normal fan of the activation polytope, and ii) combinatorially through a list of properties of associated bipartite graphs, in analogy to covector axioms of oriented matroids and tropical oriented matroids. Our findings extend and refine the connection between neural networks and tropical geometry by observing structures established in real tropical geometry, such as positive tropicalizations of hypersurfaces and tropical semialgebraic sets.

Figures (6)

• Figure 1: The three combinatorial types of generic hypersurfaces defined by tropical polynomials in two variables and $4$ terms (top), and the dual regular subdivisions of their Newton polygons (bottom), as described in \ref{['ex:hypersurfaces']}.
• Figure 2: All combinatorial possibilities of positive and negative regions of $\mathcal{T}(g\oplus h)$ for $n=m=2$, up to switching "$+$" and "$-$", as explained in \ref{['ex:signed-sectors']}. The decision boundary $\mathcal{B}(g \oslash h)$ (top) and its dual complex (bottom) are highlighted in blue. The right most column shows configurations of three positive regions and one negative region ($n=3, m=1$). The decision boundary is highlighted in orange.
• Figure 3: The activation pattern $G$ from \ref{['ex:bipartite-graphs']}, partitions $\mathcal{T}(f_\theta)$ of the input space for three different values of $\theta \in C_\mathcal{D}( G )$ and dual subdivisions of Newton polytopes $\mathop{\mathrm{Newt}}\nolimits(f_\theta)$.
• Figure 4: A coloring of the bipartite graph and the corresponding regions, as described in \ref{['ex:points-in-signed-sectors']}. The positive regions are labeled by $i_1,i_2$ and colored in green, the negative regions are labeled by $j_1,j_2$ and colored in red, and the decision boundary is highlighted in blue.
• Figure 5: The $12$ cones of correct classification from the example in the proof of \ref{['ex:classification-disconneted']}, grouped in strongly connected components.

Theorems & Definitions (68)

• Theorem 2.1
• Proposition 2.2
• proof
• Theorem 2.3
• proof
• Theorem 3.1: arora18_understandingdeepneural and zhang18_tropicalgeometrydeep
• Theorem 3.2
• proof
• Remark 3.3: $\mathcal{S}$ is not unique
• Example 4.1: Tropical Hypersurfaces and Newton Polytopes