ENFORCE: Nonlinear Constrained Learning with Adaptive-depth Neural Projection

TL;DR

ENFORCE tackles the challenge of strictly enforcing nonlinear equality constraints in neural networks by embedding a differentiable, adaptive-depth projection module (AdaNP) into the architecture. The projection is proven to be 1-Lipschitz, non-expansive, and conducive to stable gradient flow, while AdaNP adaptively increases projection depth to meet a tolerance on constraint satisfaction. The method achieves constraint feasibility with modest overhead and can substantially accelerate large-scale constrained optimization (up to ~25× faster than IPOPT) while preserving near-optimal objectives and improving predictive accuracy over unconstrained baselines. These results demonstrate a scalable, solver-free pathway to hard-constrained learning, with implications for safety-critical and physics-informed applications, and potential extensions to piecewise or inequality constraints and integration with GenAI pipelines.

Abstract

Ensuring neural networks adhere to domain-specific constraints is crucial for addressing safety and ethical concerns while also enhancing inference accuracy. Despite the nonlinear nature of most real-world tasks, existing methods are predominantly limited to affine or convex constraints. We introduce ENFORCE, a neural network architecture that uses an adaptive projection module (AdaNP) to enforce nonlinear equality constraints in the predictions. We prove that our projection mapping is 1-Lipschitz, making it well-suited for stable training. We evaluate ENFORCE on an illustrative regression task and for learning solutions to high-dimensional optimization problems in an unsupervised setting. The predictions of our new architecture satisfy $N_C$ equality constraints that are nonlinear in both the inputs and outputs of the neural network, while maintaining scalability with a tractable computational complexity of $\mathcal{O}(N_C^3)$ at training and inference time.

Figures (8)

• Figure 1: ENFORCE consists of a backbone neural network and an adaptive neural projection (AdaNP) module. The backbone network can be of every kind, such as fully connected, convolutional, or transformer architecture. AdaNP includes an adaptive number of neural projection (NP) layers, each composed of an auto-differentiation and a local projection step.
• Figure 2: Prediction comparison between ENFORCE ($\lambda_D = 0.5$, $\epsilon_T = 10^{-4}$, $\epsilon_I = 10^{-6}$) and a multilayer perceptron (MLP). ENFORCE enhances the overall accuracy and guarantees satisfaction for highly nonlinear constraints. ENFORCE consistently performs better than a standard MLP even when trained on uniformly sampled fractions of the training dataset. Interestingly, ENFORCE outperforms the MLP in data-scarce regions of the domain, which in this dataset correspond to the domain extremities (as shown in Fig. \ref{['fig:y2']}). More generally, ENFORCE also performs better under data-scarcity conditions when the models are trained on uniformly sampled fractions of the dataset (Fig. \ref{['fig:data-scarcity']}). This observation suggests that constrained learning may enhance data efficiency.
• Figure 3: ENFORCE demonstrates significantly improved convergence, achieving lower loss compared to an unconstrained MLP. Enhanced training performances are reported for the backbone network of ENFORCE even before the action of AdaNP. This effect is enabled by the simultaneous minimization of the projection displacement (in green) and the action of the AdaNP module (in yellow). Note that we report average values across multiple runs, which explains why the depth of AdaNP appears as a step function with non-integer values.
• Figure 4: Influence of constrained learning hyperparameters on the accuracy of ENFORCE on the test set (note that here we plot the inverse of the mean squared error (MSE)). The weighting factor $\lambda_D$ favors the learning process if appropriately tuned. Conversely, the training tolerance $\epsilon_T$ exhibits a small impact on performance, suggesting it can be set based on available resources. Overall, despite the choice of hyperparameters, ENFORCE is more accurate than an MLP with the same complexity, while also satisfying the underlying nonlinear constraint.
• Figure 5: Dynamic evolution of AdaNP during training and inference when different training hyperparameters are chosen. At training time, AdaNP is deeper as a smaller constraint tolerance $\epsilon_T$ is chosen.

Theorems & Definitions (12)

• Definition 1: Projection operator $\mathcal{P}$
• Definition 2: AdaNP module
• Proposition 1: Convergence of AdaNP
• Proposition 2: Uniqueness of the projection
• Proposition 3: No-worse property
• Theorem 1: Non-expansiveness of the projection operator
• Lemma 1: Gradient flow dynamics
• Definition 3: Tubular neighbourhood
• Definition 4: Reach of a manifold Federer1959_Curvaturemeasures
• Theorem 2: Uniqueness of the projection