Combining Constraint Programming Reasoning with Large Language Model Predictions

TL;DR

This paper proposes a solution by combining both approaches and embedding a Large Language Model (LLM) in Constraint Programming (CP), which presents new possibilities for enhancing text generation under constraints.

Abstract

Constraint Programming (CP) and Machine Learning (ML) face challenges in text generation due to CP's struggle with implementing "meaning'' and ML's difficulty with structural constraints. This paper proposes a solution by combining both approaches and embedding a Large Language Model (LLM) in CP. The LLM handles word generation and meaning, while CP manages structural constraints. This approach builds on GenCP, an improved version of On-the-fly Constraint Programming Search (OTFS) using LLM-generated domains. Compared to Beam Search (BS), a standard NLP method, this combined approach (GenCP with LLM) is faster and produces better results, ensuring all constraints are satisfied. This fusion of CP and ML presents new possibilities for enhancing text generation under constraints.

Figures (4)

• Figure 1: Our approach aspires to explore the in-between area. In the blue (left-hand side) region, LLM guided searches solve weakly constrained problems lu-etal-2022-neurologicbeamsearch2:2018hokamp-liu:2017 and in the green (right-hand side) region, CP-based generation tackles strongly constrained problems sprockeels-vanroy:2024expressingbonlarron-regin:2024interbonlarron-regin:2024markovbonlarron-et-al:2023perez-regin:17bpapadopoulos-roy-etal:15pachet-roy:2011.
• Figure 2: This scheme presents the integration of ML into CP performed by GenCP. It is freely inspired by Sec. 3.2.3 of Bengio et al.'s survey BENGIO-survey:2021, which introduces an architecture for ML alongside Optimization Algorithms. The similarity is highlighted because the master algorithm (here, GenCP) repeatedly queries an ML model (here, an LLM) to obtain a prediction as a subroutine. In the context of this paper, the decision (search or propagation) has an impact on the problem definition (the CSP) because it may generate new variables, domains, or constraints during the solving process. The state is the current assignment of the variables.
• Figure 3: This graph illustrates the main steps in GenCP solving.
• Figure 4: Illustrations of GenCP as a simplified framework with three variables and predictions of 3 values per LLM call, on a simple problem: generate a sentence that does not contain the letter e. For each variable, the predefined constraint $c_i$ "the letter e is forbidden" is generated. A predefined domain with one word is defined for the first variable: A. The current sentence formed by the variable "A" is not a solution ($callLLM("A")$ does not answer a period (".")), so a new empty variable $x_2$ is generated. GenCP calls the LLM with the sentence "A" to predict the domain of $x_2$. The LLM model predicts three values: [ man, house, boy ]. $c_2$ is generated, causing the domain of $x_2$ to be filtered accordingly: house is removed. $x_2$ is then assigned to boy, GenCP generates the variable $x_3$ and calls the LLM with the sentence "A boy" to predict a new domain. Unfortunately, the domain of $x_3$ is empty, either because the LLM answered an empty domain or because this domain was entirely filtered during propagation. Hence, GenCP backtracks to $x_2$ and the value of $x_2$ is changed to man. In the same fashion as before, GenCP generates the variables $x_3$, and gives "A man" to the LLM that predicts: [ drinks, and, helps ]. $c_3$ is generated, filtering helps because it contains an e. The process continues until the domain of the next predicted variable contains a period (a solution is found) or the solver fails.