OptiTree: Hierarchical Thoughts Generation with Tree Search for LLM Optimization Modeling

TL;DR

OptiTree presents a novel approach to optimization modeling with large language models by organizing modeling knowledge into a hierarchical modeling tree and guiding modeling through tree search. By adaptively decomposing complex OR problems into subproblems and synthesizing global modeling thoughts from hierarchical patterns, it achieves significant improvements over state-of-the-art baselines on challenging benchmarks. The method generalizes across multiple LLMs and datasets, and the authors release code to enable reproducibility. Ablation studies reveal the critical roles of tree search and modeling thoughts, and highlight the benefits of deeper search and statement-based subproblem identification. Overall, OptiTree offers a scalable, reasoning-driven path to more accurate and reliable LLM-based optimization modeling in operations research.

Abstract

Optimization modeling is one of the most crucial but technical parts of operations research (OR). To automate the modeling process, existing works have leveraged large language models (LLMs), prompting them to break down tasks into steps for generating variables, constraints, and objectives. However, due to the highly complex mathematical structures inherent in OR problems, standard fixed-step decomposition often fails to achieve high performance. To address this challenge, we introduce OptiTree, a novel tree search approach designed to enhance modeling capabilities for complex problems through adaptive problem decomposition into simpler subproblems. Specifically, we develop a modeling tree that organizes a wide range of OR problems based on their hierarchical problem taxonomy and complexity, with each node representing a problem category and containing relevant high-level modeling thoughts. Given a problem to model, we recurrently search the tree to identify a series of simpler subproblems and synthesize the global modeling thoughts by adaptively integrating the hierarchical thoughts. Experiments show that OptiTree significantly improves the modeling accuracy compared to the state-of-the-art, achieving over 10\% improvements on the challenging benchmarks. The code is released at https://github.com/MIRALab-USTC/OptiTree/tree/main.

Figures (10)

• Figure 1: The Upper: Existing prompt-based methods construct variables, constraints and objectives sequentially. The Lower: Our OptiTree leverages tree search for subproblem decomposition and uses the modeling thoughts of subproblems to enhance the modeling process.
• Figure 2: The observations. Left: We find most errors come from the incorrect variable definition in the Medium and Hard problems. Middle: We find the most complex problems have subproblems of standard OR problems, and LLMs are effective in identifying the subproblems. Right: We observe an improvement in modeling accuracy using subproblem decomposition.
• Figure 3: The overview of OptiTree. (1) OptiTree constructs a modeling tree for subproblem decomposition and stores the modeling thoughts for subproblems in the tree nodes. (2) OptiTree leverages tree search to identify the subproblems and construct modeling thoughts by integrating the subproblems' thoughts. (3) We update the modeling tree to ensure its scalability and reliability.
• Figure 4: Left: OptiTree can identify subproblems for a wide range of problems, including the challenging datasets. Right: We manually check the accuracy of the subproblem decomposition.
• Figure 5: Ablation studies on the tree search (left) and modeling thoughts (right). The results demonstrate the critical roles of these two components.

Theorems & Definitions (4)

• Definition 4.1
• Proposition 4.2
• Lemma F.1: Transitivity of $\subseteq_{\mathcal{S}}$
• proof