Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations

TL;DR

This work expands the theoretical foundation of monotone neural networks by showing that saturating activations with alternating saturation sides enable universal monotone approximation with a constant-depth MLP, even when weight constraints are relaxed or reversed to non-positive. It establishes a key link between activation saturation and weight sign, proving non-positive constrained networks (and activation-reflection variants) can be universal while non-positive-constrained ReLU-like networks offer advantages over traditional non-negative formulations. The authors then introduce a practical activation-switch parametrization that obviates the need for weight reparameterization, improving initialization robustness and training stability, and validate the approach with experiments on multiple monotone tasks where it matches or surpasses state-of-the-art baselines. Collectively, the results broaden architectural options for monotone networks while providing both rigorous theory and actionable training techniques.

Abstract

Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which pose well-known optimization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, we show an equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the empirical effectiveness observed in previous works while leading to possible architectural simplification. Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability. Experimental evaluation reinforces the validity of the theoretical results, showing that our novel approach compares favourably to traditional monotonic architectures.

Figures (15)

• Figure 1: Monotone MLPs with weight-constraint and bounded activations (pink) and our proposed approach based on ReLU (blue). The former can only represent bounded functions and, thus, cannot extrapolate the data trend, which is important in many domains, such as time-series analysis and predictive maintenance.
• Figure 2: Constructions of Heavyside function using a composition of ReLU and its point reflection ReLU' to obtain ${\text{ReLU}(\text{ReLU}'(\alpha x -0.5) + 1) = \text{ReLU}'(\text{ReLU}(\alpha x + 0.5) - 1)}$
• Figure 3: Example of representable functions at layer $1$.
• Figure 4: Example of representable functions at layer $2$.
• Figure 5: Example of representable functions at layer $3$.

Theorems & Definitions (25)

• Remark 3.1
• Proposition 3.2
• Definition 3.3
• Proposition 3.4
• Theorem 3.5
• Lemma 3.6
• proof
• Lemma 3.7
• proof
• proof : Proof of \ref{['thm:universal_approx']}