An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models

TL;DR

The paper tackles the challenge of validating LLM-generated optimization models derived from natural-language descriptions by introducing an automated, agent-based validation framework. It adapts software testing principles, notably mutation testing, to optimization models through a four-agent workflow that builds a problem-level testing API, generates unit tests, creates an auxiliary optimization model, and injects targeted mutations to probe test efficacy. Empirical results on the NLP4LP dataset show high mutation coverage and robust auxiliary-model validation, with external-model testing indicating practical usefulness though with some false positives. The framework promises improved reliability for automated optimization modeling and offers pathways for refining mutation strategies and testing on more complex, real-world problems.

Abstract

Recently, using Large Language Models (LLMs) to generate optimization models from natural language descriptions has became increasingly popular. However, a major open question is how to validate that the generated models are correct and satisfy the requirements defined in the natural language description. In this work, we propose a novel agent-based method for automatic validation of optimization models that builds upon and extends methods from software testing to address optimization modeling . This method consists of several agents that initially generate a problem-level testing API, then generate tests utilizing this API, and, lastly, generate mutations specific to the optimization model (a well-known software testing technique assessing the fault detection power of the test suite). In this work, we detail this validation framework and show, through experiments, the high quality of validation provided by this agent ensemble in terms of the well-known software testing measure called mutation coverage.

Figures (6)

• Figure 1: LP - What we want to cover.
• Figure 2: Flow and expected outcomes. (a) Test suite generation flow. (b) Outcome matrix for combining test-suite quality (Good/Bad), model correctness (Good/Bad), and mutation validity (Good/Bad): a good suite with a good model should pass on the base model and typically fail on a mutated model; “?” entries indicate cases where outcomes depend on specifics (e.g., weak suites or invalid mutations).
• Figure 3: Agents and their I/O. (a) Business Interface Generator — inputs: problem description, interface template, and generation instructions; output: business-interface code. (b) Unit Tests Generator — inputs: problem description, business interface, and test-generation instructions; output: unit-test suite.
• Figure 4: Agents and their I/O. (a) Optimization Model Generator — inputs: problem description, business interface, unit tests, and generation instructions; output: optimization model code. (b) Mutated Optimization Model Agent — inputs: problem description, business interface, baseline model, and mutation rules; output: mutated optimization model.
• Figure 5: Agents and their I/O. (a) Tests Adjuster — input: problem description, optimization model, and original unit tests; output: adjusted API-aligned test suite. (b) Test Adjuster Agent — same I/O represented as a process diagram.