Preference Elicitation for Step-Wise Explanations in Logic Puzzles

TL;DR

The paper addresses how to elicit and learn user preferences for step-wise explanations in constraint programming by extending Constructive Preference Elicitation (CPE) to explanation steps. It introduces MACHOP, a query-generation strategy that combines non-domination constraints with UCB-inspired diversification and proposes robust normalization schemes to stabilize learning across diverse sub-objectives. Through experiments on Sudoku and Logic-Grid puzzles with simulated and real users, MACHOP consistently yields higher-quality explanations and faster convergence than baseline methods, demonstrating practical viability for interactive explainable CP. Overall, the work advances user-centered explainability for complex CSP explanations and provides actionable methods for learning, generating, and evaluating preference-guided explanations.

Abstract

Step-wise explanations can explain logic puzzles and other satisfaction problems by showing how to derive decisions step by step. Each step consists of a set of constraints that derive an assignment to one or more decision variables. However, many candidate explanation steps exist, with different sets of constraints and different decisions they derive. To identify the most comprehensible one, a user-defined objective function is required to quantify the quality of each step. However, defining a good objective function is challenging. Here, interactive preference elicitation methods from the wider machine learning community can offer a way to learn user preferences from pairwise comparisons. We investigate the feasibility of this approach for step-wise explanations and address several limitations that distinguish it from elicitation for standard combinatorial problems. First, because the explanation quality is measured using multiple sub-objectives that can vary a lot in scale, we propose two dynamic normalization techniques to rescale these features and stabilize the learning process. We also observed that many generated comparisons involve similar explanations. For this reason, we introduce MACHOP (Multi-Armed CHOice Perceptron), a novel query generation strategy that integrates non-domination constraints with upper confidence bound-based diversification. We evaluate the elicitation techniques on Sudokus and Logic-Grid puzzles using artificial users, and validate them with a real-user evaluation. In both settings, MACHOP consistently produces higher-quality explanations than the standard approach.

Figures (5)

• Figure 1: Two Sudoku explanation steps that explain $\mathtt{cell[7,7] = 6}$ (in green). Used facts $\mathcal{E}$ are in yellow, while used constraints $\mathcal{S}$ are in blue. The table provides an example of the mapping from explanations to features.
• Figure 2: Average relative regret for explanation sequences.
• Figure 3: Relative regret for the different diversification strategies for Sudoku and LGPs.
• Figure 4: Two Sudoku explanation steps that explain $\mathtt{cell[7,7] = 6}$ (in green). Used facts $\mathcal{E}$ are in yellow, while used constraints $\mathcal{S}$ are in blue. The table provides the exact mapping used from explanations to features.
• Figure :

Theorems & Definitions (8)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Definition 7
• Definition 8