Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inverse Optimization Without Inverse Optimization: Direct Solution Prediction with Transformer Models

Macarena Navarro, Willem-Jan van Hoeve, Karan Singh

TL;DR

The paper tackles learning solutions to discrete optimization problems with unknown objectives or constraints by replacing traditional inverse optimization with a structured-prediction approach that uses transformer-based sequence-to-sequence models. A DFA-based constraint reasoning module masks the decoder to ensure feasibility, enabling end-to-end learning of latent objective and constraint structure from data. Across knapsack, bipartite matching, and single-machine scheduling, the transformer framework consistently achieves high-quality, feasible solutions with far faster inference than IO and outperforming LSTM baselines, even under data corruption and varying training sizes. The approach offers a scalable, robust alternative for decision problems where exact objective forms or implicit constraints are unknown, relying on monotone constraint systems and rich historical data to capture complex latent structure.

Abstract

We present an end-to-end framework for generating solutions to combinatorial optimization problems with unknown components using transformer-based sequence-to-sequence neural networks. Our framework learns directly from past solutions and incorporates the known components, such as hard constraints, via a constraint reasoning module, yielding a constrained learning scheme. The trained model generates new solutions that are structurally similar to past solutions and are guaranteed to respect the known constraints. We apply our approach to three combinatorial optimization problems with unknown components: the knapsack problem with an unknown reward function, the bipartite matching problem with an unknown objective function, and the single-machine scheduling problem with release times and unknown precedence constraints, with the objective of minimizing average completion time. We demonstrate that transformer models have remarkably strong performance and often produce near-optimal solutions in a fraction of a second. They can be particularly effective in the presence of more complex underlying objective functions and unknown implicit constraints compared to an LSTM-based alternative and inverse optimization.

Inverse Optimization Without Inverse Optimization: Direct Solution Prediction with Transformer Models

TL;DR

The paper tackles learning solutions to discrete optimization problems with unknown objectives or constraints by replacing traditional inverse optimization with a structured-prediction approach that uses transformer-based sequence-to-sequence models. A DFA-based constraint reasoning module masks the decoder to ensure feasibility, enabling end-to-end learning of latent objective and constraint structure from data. Across knapsack, bipartite matching, and single-machine scheduling, the transformer framework consistently achieves high-quality, feasible solutions with far faster inference than IO and outperforming LSTM baselines, even under data corruption and varying training sizes. The approach offers a scalable, robust alternative for decision problems where exact objective forms or implicit constraints are unknown, relying on monotone constraint systems and rich historical data to capture complex latent structure.

Abstract

We present an end-to-end framework for generating solutions to combinatorial optimization problems with unknown components using transformer-based sequence-to-sequence neural networks. Our framework learns directly from past solutions and incorporates the known components, such as hard constraints, via a constraint reasoning module, yielding a constrained learning scheme. The trained model generates new solutions that are structurally similar to past solutions and are guaranteed to respect the known constraints. We apply our approach to three combinatorial optimization problems with unknown components: the knapsack problem with an unknown reward function, the bipartite matching problem with an unknown objective function, and the single-machine scheduling problem with release times and unknown precedence constraints, with the objective of minimizing average completion time. We demonstrate that transformer models have remarkably strong performance and often produce near-optimal solutions in a fraction of a second. They can be particularly effective in the presence of more complex underlying objective functions and unknown implicit constraints compared to an LSTM-based alternative and inverse optimization.
Paper Structure (36 sections, 3 theorems, 16 equations, 12 figures, 20 tables, 1 algorithm)

This paper contains 36 sections, 3 theorems, 16 equations, 12 figures, 20 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formalization and Latent Structure
  3. Problem Definition
  4. Types of Latent Problem Structure
  5. Structured Prediction with Transformers
  6. Sequence-to-Sequence Models
  7. Constraint Reasoning
  8. Transformer Architecture with DFA-Based Constraint Reasoning
  9. Inductive Bias Considerations
  10. Application to Three Combinatorial Problems
  11. Knapsack Problem with an Unknown Reward Function
  12. Bipartite Matching Problem with an Unknown Objective Function
  13. Single-Machine Scheduling Problem with Release Times and Unknown Precedence Constraints
  14. Algorithmic Benchmarks
  15. Inverse Optimization Approach
  16. ...and 21 more sections

Key Result

Proposition 3.1

Algorithm alg:sampling returns a solution $\sigma \in \mathcal{X}(u)$ with probability 1 assuming that each feasible transition is sampled with positive probability and $\mathcal{X}(u)$ is non-empty.

Figures (12)

  • Figure 1: Transformer encoder-decoder architecture with DFA-guided constraint masking.
  • Figure 2: Optimality gaps (%) for the linear instances of the knapsack problem.
  • Figure 3: Optimality gaps (%) for the quadratic instances of the knapsack problem.
  • Figure 4: Optimality gaps (%) for the quadratic bipartite matching problem.
  • Figure 5: Optimality gaps (%) for instances of length 20 for the single-machine scheduling problem.
  • ...and 7 more figures

Theorems & Definitions (9)

  • Definition 3.1
  • Proposition 3.1
  • proof : Proof.
  • Definition 3.2
  • Definition 3.3
  • Proposition 3.2
  • proof : Proof.
  • Proposition 3.3
  • proof : Proof.