Lagrange Multipliers and Duality with Applications to Constrained Support Vector Machine

TL;DR

The paper develops a rigorous Lagrange multiplier and duality framework for nonsmooth convex optimization in Hilbert spaces via the quasi-relative interior, deriving enhanced KKT conditions and strong duality under Slater-type assumptions. It then generalizes SVM by imposing a geometric constraint or regularizer on the separating hyperplane and analyzes this constrained SVM through Lagrangian duality, proximal mappings, and Moreau envelopes. The work provides explicit dual formulations for hard-margin, soft-margin, and regularized SVM variants, alongside convergent subgradient and primal-dual methods, and reveals a squared-distance term to the constraint set in the dual. These contributions offer principled, scalable approaches to constrained SVMs with provable optimality conditions and practical algorithms for large-scale classification in Hilbert spaces, including kernelized settings.

Abstract

In this paper, we employ the concept of quasi-relative interior to analyze the method of Lagrange multipliers and establish strong Lagrangian duality for nonsmooth convex optimization problems in Hilbert spaces. Then, we generalize the classical support vector machine (SVM) model by incorporating a new geometric constraint or a regularizer on the separating hyperplane, serving as a regularization mechanism for the SVM. This new SVM model is examined using Lagrangian duality and other convex optimization techniques in both theoretical and numerical aspects via a new subgradient algorithm as well as a primal-dual method.

Theorems & Definitions (62)

• Theorem 2.1
• Theorem 2.2
• Theorem 2.3
• Lemma 2.4
• proof
• Theorem 2.5
• proof
• Lemma 3.1
• proof
• Theorem 3.2