Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing

Francesca Lanzillotta, Chiara Albisani, Davide Pucci, Daniele Baracchi, Alessandro Piva, Matteo Lapucci

TL;DR

The paper addresses learning problems where per-sample outputs must satisfy strict constraints, formalizing this as min_w L(w) = ∑_{i=1}^N ℓ(w; x^i,y^i) with c(w; x^i) ≤ B for all i. It introduces a sequential quadratic penalty method with an inexact stochastic inner loop: outer iterations solve min_x P_{τ_k}(x) where P_τ(x) = f(x) + (τ/2) ∑_{i=1}^m max{0,g_i(x)}^2, and inner SGD achieves E[||∇P_{τ_k}(x^k)||] ≤ ε_k with τ_k → ∞, ε_k → 0. Theoretical contributions include finite-term guarantees for the inner solver (T_k formula) and outer-loop convergence: ∇P_{τ_k}(x^k) → 0 in probability, with almost sure subsequential convergence to a KKT point under E-LICQ. Empirically, the framework is validated on image-processing tasks, notably MNIST with a reconstruction constraint and a PSNR-based constraint for medical image watermarking, demonstrating effective constraint satisfaction without sacrificing primary task performance. Overall, the work provides a principled, convergent optimization paradigm for constrained learning in contexts where per-sample guarantees are semantically meaningful and practically impactful.

Abstract

In many learning tasks, certain requirements on the processing of individual data samples should arguably be formalized as strict constraints in the underlying optimization problem, rather than by means of arbitrary penalties. We show that, in these scenarios, learning can be carried out exploiting a sequential penalty method that allows to properly deal with constraints. The proposed algorithm is shown to possess convergence guarantees under assumptions that are reasonable in deep learning scenarios. Moreover, the results of experiments on image processing tasks show that the method is indeed viable to be used in practice.

Sample-wise Constrained Learning via a Sequential Penalty Approach with Applications in Image Processing

TL;DR

The paper addresses learning problems where per-sample outputs must satisfy strict constraints, formalizing this as min_w L(w) = ∑_{i=1}^N ℓ(w; x^i,y^i) with c(w; x^i) ≤ B for all i. It introduces a sequential quadratic penalty method with an inexact stochastic inner loop: outer iterations solve min_x P_{τ_k}(x) where P_τ(x) = f(x) + (τ/2) ∑_{i=1}^m max{0,g_i(x)}^2, and inner SGD achieves E[||∇P_{τ_k}(x^k)||] ≤ ε_k with τ_k → ∞, ε_k → 0. Theoretical contributions include finite-term guarantees for the inner solver (T_k formula) and outer-loop convergence: ∇P_{τ_k}(x^k) → 0 in probability, with almost sure subsequential convergence to a KKT point under E-LICQ. Empirically, the framework is validated on image-processing tasks, notably MNIST with a reconstruction constraint and a PSNR-based constraint for medical image watermarking, demonstrating effective constraint satisfaction without sacrificing primary task performance. Overall, the work provides a principled, convergent optimization paradigm for constrained learning in contexts where per-sample guarantees are semantically meaningful and practically impactful.

Abstract

In many learning tasks, certain requirements on the processing of individual data samples should arguably be formalized as strict constraints in the underlying optimization problem, rather than by means of arbitrary penalties. We show that, in these scenarios, learning can be carried out exploiting a sequential penalty method that allows to properly deal with constraints. The proposed algorithm is shown to possess convergence guarantees under assumptions that are reasonable in deep learning scenarios. Moreover, the results of experiments on image processing tasks show that the method is indeed viable to be used in practice.
Paper Structure (10 sections, 6 theorems, 47 equations, 7 figures, 2 tables)

This paper contains 10 sections, 6 theorems, 47 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. A Sequential Penalty Approach with Inexact Stochastic Solver
  4. Convergence Analysis
  5. Finite termination of the inner solver
  6. Outer Loop Convergence
  7. Computational Experiments
  8. Preliminary algorithm study
  9. A case study: Medical Image Watermarking
  10. Conclusions

Key Result

Theorem 1

If $x^\ast$ is a local minimum of problem eq:gen_prob and the LICQ holds at $x^\ast$, then $x^\ast$ satisfies the KKT conditions.

Figures (7)

  • Figure 1: Network architecture for the toy problem: input images go through two fully connected layers mapping to a 20-dimensional encoding, then forwarded to distinct branches to get class predictions and the reconstructed images.
  • Figure 2: Train and test densities of the reconstruction loss for the sequential penalty method and for the fixed regularization method.
  • Figure 3: Train and test accuracy and percentage of satisfied constraints during the training with the sequential penalty method, the fixed regularization approach, and, as a baseline, only considering the classification loss.
  • Figure 4: HiDDeN overview. Our proposed solution replaces $\mathcal{L}_I$ with a PSNR-based constraint.
  • Figure 5: Training (left) and validation (right) curves of the message loss $\mathcal{L}_M \left( M_{in}, M_{out} \right)$ and $\text{PSNR} \left( I_{co}, I_{en} \right)$ for all model configurations.
  • ...and 2 more figures

Theorems & Definitions (16)

  • Definition 1: Karush--Kuhn--Tucker (KKT) conditions
  • Definition 2: Linear Independence Constraint Qualification (LICQ)
  • Definition 3: Extended Linear Independence Constraint Qualification (E-LICQ)
  • Theorem 1
  • Definition 4
  • Definition 5: schmidt2013fast
  • Definition 6: Convergence in probability
  • Definition 7: Convergence almost surely
  • Lemma 1: durrett2019probability
  • Lemma 2: Markov's inequality
  • ...and 6 more