Convexity in ReLU Neural Networks: beyond ICNNs?

TL;DR

The paper tackles when ReLU networks implement convex functions and introduces a path-lifting framework to derive necessary and sufficient convexity conditions. It proves that 1-hidden-layer networks coincide with ICNNs, but 2-hidden-layer networks can realize convex CPWL functions beyond ICNNs and provides explicit counterexamples. A general DAG-ReLU analysis with path-lifting yields both necessary and (under mild non-degeneracy) sufficient convexity criteria, plus a practical numerical algorithm to check convexity on networks with many affine regions. The findings reveal substantial expressive life beyond ICNNs, enabling exact convexity tests and suggesting regularizers to promote convexity during training, with implications for imaging, proximal methods, and optimal transport. These results bridge CPWL convexity theory with modern neural architectures, highlighting both theoretical and practical gains in guaranteeing convex priors in learned models.

Abstract

Convex functions and their gradients play a critical role in mathematical imaging, from proximal optimization to Optimal Transport. The successes of deep learning has led many to use learning-based methods, where fixed functions or operators are replaced by learned neural networks. Regardless of their empirical superiority, establishing rigorous guarantees for these methods often requires to impose structural constraints on neural architectures, in particular convexity. The most popular way to do so is to use so-called Input Convex Neural Networks (ICNNs). In order to explore the expressivity of ICNNs, we provide necessary and sufficient conditions for a ReLU neural network to be convex. Such characterizations are based on product of weights and activations, and write nicely for any architecture in the path-lifting framework. As particular applications, we study our characterizations in depth for 1 and 2-hidden-layer neural networks: we show that every convex function implemented by a 1-hidden-layer ReLU network can be also expressed by an ICNN with the same architecture; however this property no longer holds with more layers. Finally, we provide a numerical procedure that allows an exact check of convexity for ReLU neural networks with a large number of affine regions.

Figures (10)

• Figure 1: Architecture of a $\mathtt{SkipMLP}$ network. $\mathtt{ICNN}$ imposes weights $W_2, \ldots, W_{L-1}, w_L$ to be non-negative entry-wise.
• Figure 2: DAG $G$ and extracted subgraphs $G^{\to\nu}$ and $G^{\nu\to}$.
• Figure 3: Number of convex $\mathop{\mathrm{ReLU}}\nolimits$ networks among $10^{4}$ draws. (Left): convex $\mathop{\mathrm{ReLU}}\nolimits$ networks. (Right): ICNNs. Architecture $\mathbf n = (n_1, n_2)$. The probability of sampling a convex function is higher than that of obtaining an ICNN, with a gap which becomes larger as the width of the layers increases. It confirms that ICNN constraints do restrict the set of implementable convex functions at a given architecture.
• Figure 4: The regions $R_k$ and $R_\ell$ are neighbouring, but not $R_j$ and $R_\ell$. A frontier $F_{k,\ell}$ is the interior of the facet of dimension $d-1$ which separates two distinct neighbouring regions. Convexity is a local property that has to be studied on a ball around $x$.
• Figure 5: Left: the segment does not cross any pathological points. Right: the red point is pathological: it belongs to more than two neighboring regions.

Theorems & Definitions (62)

• Proposition 2.1
• Definition 3.1: Polyhedral partition, gorokhovik1994piecewise
• Definition 3.2: CPWL function
• Definition 3.3: Compatible partition
• Definition 3.4: Neighboring regions
• Definition 3.5: Frontiers
• Remark 3.6
• Proposition 3.7
• Definition 4.1: Isolated neuron
• Lemma 4.2: Non-negativity of the last layer