Semantic Objective Functions: A distribution-aware method for adding logical constraints in deep learning

TL;DR

The paper addresses embedding logical constraints into probabilistic predictors by introducing Semantic Objective Functions (SOFs), a distribution-aware loss framework that uses information-geometric distances to align model outputs with external constraint distributions. It formalizes a constraint distribution from propositional or first-order logic constraints and supports finite and continuous domains, using the Fisher-Rao distance or KL divergence to regularize learning and enable constraint transfer from expert models. The approach extends weighted model counting to distributions and supports constraint learning, knowledge distillation, and constraint-dependent formulations, with both internal and external constraint scenarios. Empirical results on constraint learning, one-hot classification tasks with logic constraints, and preliminary first-order logic experiments demonstrate that SOFs can learn and enforce constraints effectively and can yield parameter-efficient KD while maintaining competitive accuracy.

Abstract

Issues of safety, explainability, and efficiency are of increasing concern in learning systems deployed with hard and soft constraints. Symbolic Constrained Learning and Knowledge Distillation techniques have shown promising results in this area, by embedding and extracting knowledge, as well as providing logical constraints during neural network training. Although many frameworks exist to date, through an integration of logic and information geometry, we provide a construction and theoretical framework for these tasks that generalize many approaches. We propose a loss-based method that embeds knowledge-enforces logical constraints-into a machine learning model that outputs probability distributions. This is done by constructing a distribution from the external knowledge/logic formula and constructing a loss function as a linear combination of the original loss function with the Fisher-Rao distance or Kullback-Leibler divergence to the constraint distribution. This construction includes logical constraints in the form of propositional formulas (Boolean variables), formulas of a first-order language with finite variables over a model with compact domain (categorical and continuous variables), and in general, likely applicable to any statistical model that was pretrained with semantic information. We evaluate our method on a variety of learning tasks, including classification tasks with logic constraints, transferring knowledge from logic formulas, and knowledge distillation from general distributions.