Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Positivity sets of hinge functions

Josef Schicho, Ayush Kumar Tewari, Audie Warren

TL;DR

The paper investigates which subsets of $\mathbb{R}^d$ can be realized as positivity sets of hinge functions, i.e., outputs positive on a region classified by a single hidden layer ReLU network with skip connections. It provides a complete geometric characterization for open polytopal cones in $\mathbb{R}^2$, namely that a cone $C$ is a hinge-positivity set iff $\langle G_C\rangle \cap \mathrm{Conv}(R_C)=\emptyset$, and establishes a local necessary condition in higher dimensions via local cones $C_q$. It then constructs hinge functions realizing all cones satisfying the local condition by building a symmetric, positively homogeneous, piecewise-linear function $s$ together with a linear term $\boldsymbol{e}\cdot x$ so that $f(x)=s(x)+\boldsymbol{e}\cdot x$ has the desired positivity set, and provides explicit examples (e.g., a triangle and a multi-component region) and a counterexample (the 3-fan) to illustrate the local-to-global distinctions. Overall, the work advances understanding of the representable positivity regions for single-hidden-layer ReLU networks and offers a concrete interpolation framework for constructing hinge functions with prescribed positivity sets.

Abstract

In this paper we investigate which subsets of the real plane are realisable as the set of points on which a one-layer ReLU neural network takes a positive value. In the case of cones we give a full characterisation of such sets. Furthermore, we give a necessary condition for any subset of $\mathbb R^d$. We give various examples of such one-layer neural networks.

Positivity sets of hinge functions

TL;DR

The paper investigates which subsets of can be realized as positivity sets of hinge functions, i.e., outputs positive on a region classified by a single hidden layer ReLU network with skip connections. It provides a complete geometric characterization for open polytopal cones in , namely that a cone is a hinge-positivity set iff , and establishes a local necessary condition in higher dimensions via local cones . It then constructs hinge functions realizing all cones satisfying the local condition by building a symmetric, positively homogeneous, piecewise-linear function together with a linear term so that has the desired positivity set, and provides explicit examples (e.g., a triangle and a multi-component region) and a counterexample (the 3-fan) to illustrate the local-to-global distinctions. Overall, the work advances understanding of the representable positivity regions for single-hidden-layer ReLU networks and offers a concrete interpolation framework for constructing hinge functions with prescribed positivity sets.

Abstract

In this paper we investigate which subsets of the real plane are realisable as the set of points on which a one-layer ReLU neural network takes a positive value. In the case of cones we give a full characterisation of such sets. Furthermore, we give a necessary condition for any subset of . We give various examples of such one-layer neural networks.

Paper Structure

This paper contains 10 sections, 6 theorems, 12 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Statement of main results
  4. Valleys, mountains, and local gradients
  5. Examples
  6. A local condition for positivity sets
  7. Other direction of Theorem \ref{['thm:main']}
  8. Constructing values of a symmetric function along rays
  9. Case analysis for the interpolation
  10. Verifying the positivity set of the constructed hinge function

Key Result

Theorem 1

Let $C \subseteq \mathbb R^2$ be a polytopal cone. Then $C$ is the positivity set of a hinge function if and only if

Figures (9)

  • Figure 1: A hinge function and its corresponding positivity set. The hinge function is given by $h(x,y) =x-1 +|x-1|+|y|-|y-x|.$
  • Figure 2: The point $q$ in the blue set $P$, and its local cone $C_q$.
  • Figure 3: The hinge function $h$ from Figure \ref{['fig:hingeandpositive']} and its mountain/valley arrangement. The mountain is brown and the valleys are green.
  • Figure 4: The hinge function $h$ whose positivity set is a triangle.
  • Figure 5: A hinge function whose positivity set has four connected components.
  • ...and 4 more figures

Theorems & Definitions (11)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • proof
  • proof : Proof of Theorem \ref{['thm:localcond']}
  • Example 1
  • Proposition 1
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 1 more