A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints

TL;DR

This work introduces pseudo-semantic loss, a local, pseudolikelihood-based neuro-symbolic objective designed for autoregressive models to satisfy logical constraints without requiring exact computation over the full output distribution. By centering a fully factorized approximation around a model sample and leveraging smooth, decomposable logical circuits, the method yields tractable, differentiable objectives that preserve fidelity near the sample. Empirically, it improves structured-output generation in Sudoku and Warcraft shortest-path tasks and achieves notable detoxification effects for GPT-2, with reduced toxicity while maintaining language-model quality. The approach offers a scalable, general framework for integrating hard symbolic constraints into neural autoregressive models, with practical impact on trustworthy AI systems and constrained generation.

Abstract

Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. This often requires maximizing the likelihood of a symbolic constraint w.r.t the neural network's output distribution. Such output distributions are typically assumed to be fully-factorized. This limits the applicability of neuro-symbolic learning to the more expressive autoregressive distributions, e.g., transformers. Under such distributions, computing the likelihood of even simple constraints is #P-hard. Instead of attempting to enforce the constraint on the entire output distribution, we propose to do so on a random, local approximation thereof. More precisely, we optimize the likelihood of the constraint under a pseudolikelihood-based approximation centered around a model sample. Our approximation is factorized, allowing the reuse of solutions to sub-problems, a main tenet for efficiently computing neuro-symbolic losses. Moreover, it is a local, high-fidelity approximation of the likelihood, exhibiting low entropy and KL-divergence around the model sample. We evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation, and observe that we greatly improve upon the base model's ability to predict logically-consistent outputs. We also evaluate on the task of detoxifying large language models. Using a simple constraint disallowing a list of toxic words, we are able to steer the model's outputs away from toxic generations, achieving SoTA detoxification compared to previous approaches.

Figures (4)

• Figure 1: Our approach in a nutshell. Given a data point $x$, we approximate the likelihood of the constraint $\alpha$ (area shaded in pink) with the pseudolikelihood (shown in gray) of the constraint in the neighborhood of a sample (denoted ${\color{red}{\footnotesize{\times}}}$), where $m(\alpha)$ denotes the region of the constraint support.
• Figure 2: An example of our pipeline. (Left) We start by sampling an assignment from the model $p_{\boldsymbol{\theta}\xspace}$. Our goal is to compute the pseudolikelihood of the model sample— the product of the sample's conditionals. We start by expanding the model sample to include all samples that are a Hamming distance of $1$ away from the sample. We proceed by (batch) evaluating the samples through the model, obtaining the joint probability of each sample. We then normalize along each column, obtaining the conditionals. (Right) A logical circuit encoding constraint $(\text{Cat} \implies \text{Animal}) \land (\text{Dog} \implies \text{Animal})$, with variable $A$ mapping to Cat, variable $B$ mapping to dog and variable $C$ mapping to Animal. To compute the pseudolikelihood of the constraint in the neighborhood of the sample $abc$, we feed the computed conditional at the corresponding literals. We push the probabilities upwards, taking products at AND nodes and sums at OR nodes. The number accumulated at the root of the circuit is the pseudolikelihood of the constraint in the neighborhood of the sample $abc$.
• Figure 3: Example inputs and groundtruth labels for two of the three tasks considered in our experimental evaluation. (Left) Example Warcraft terrain map and a possible (non-unique) minimum-cost shortest path. (Right) Example Sudoku puzzle and its corresponding (unique) solution.
• Figure : $\mathcal{L}^{\mathsf{SL}}_{\mathsf{pseudo}}(\alpha; p_{\boldsymbol{\theta}\xspace}$)

Theorems & Definitions (1)

• Definition 3.1: Pseudo-Semantic Loss