Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solving

Ankur Sinha, Shobhit Arora, Dhaval Pujara

TL;DR

AutoOpt tackles the challenge of solving optimization problems described in images by introducing AutoOpt-11k, a large, diverse dataset of over 11k optimization formulations annotated with LaTeX and, for a subset, PYOMO scripts. The framework comprises three modules: M1 for image-to-LaTeX via a hybrid ResNet-101 and Swin Transformer encoder with a mBART decoder, M2 for LaTeX-to-PYOMO using a fine-tuned DeepSeek-Coder, and M3 for solving the resulting PYOMO scripts with a bilevel optimization-based decomposition (BOBD) enhanced by LR-VCM variable classification. Experimental results show M1 outperforms large LLMs on MER metrics (BLEU/CER), M2 achieves strong code-generation accuracy for PYOMO, and M3 delivers superior optimization performance across a suite of test problems compared with interior-point and genetic algorithms. The combined AutoOpt pipeline demonstrates high reliability and is poised to accelerate research at the intersection of computer vision, NLP, and mathematical optimization by enabling automatic extraction and solving of optimization problems directly from images.

Abstract

This study presents AutoOpt-11k, a unique image dataset of over 11,000 handwritten and printed mathematical optimization models corresponding to single-objective, multi-objective, multi-level, and stochastic optimization problems exhibiting various types of complexities such as non-linearity, non-convexity, non-differentiability, discontinuity, and high-dimensionality. The labels consist of the LaTeX representation for all the images and modeling language representation for a subset of images. The dataset is created by 25 experts following ethical data creation guidelines and verified in two-phases to avoid errors. Further, we develop AutoOpt framework, a machine learning based automated approach for solving optimization problems, where the user just needs to provide an image of the formulation and AutoOpt solves it efficiently without any further human intervention. AutoOpt framework consists of three Modules: (i) M1 (Image_to_Text)- a deep learning model performs the Mathematical Expression Recognition (MER) task to generate the LaTeX code corresponding to the optimization formulation in image; (ii) M2 (Text_to_Text)- a small-scale fine-tuned LLM generates the PYOMO script (optimization modeling language) from LaTeX code; (iii) M3 (Optimization)- a Bilevel Optimization based Decomposition (BOBD) method solves the optimization formulation described in the PYOMO script. We use AutoOpt-11k dataset for training and testing of deep learning models employed in AutoOpt. The deep learning model for MER task (M1) outperforms ChatGPT, Gemini and Nougat on BLEU score metric. BOBD method (M3), which is a hybrid approach, yields better results on complex test problems compared to common approaches, like interior-point algorithm and genetic algorithm.

AutoOpt: A Dataset and a Unified Framework for Automating Optimization Problem Solving

TL;DR

AutoOpt tackles the challenge of solving optimization problems described in images by introducing AutoOpt-11k, a large, diverse dataset of over 11k optimization formulations annotated with LaTeX and, for a subset, PYOMO scripts. The framework comprises three modules: M1 for image-to-LaTeX via a hybrid ResNet-101 and Swin Transformer encoder with a mBART decoder, M2 for LaTeX-to-PYOMO using a fine-tuned DeepSeek-Coder, and M3 for solving the resulting PYOMO scripts with a bilevel optimization-based decomposition (BOBD) enhanced by LR-VCM variable classification. Experimental results show M1 outperforms large LLMs on MER metrics (BLEU/CER), M2 achieves strong code-generation accuracy for PYOMO, and M3 delivers superior optimization performance across a suite of test problems compared with interior-point and genetic algorithms. The combined AutoOpt pipeline demonstrates high reliability and is poised to accelerate research at the intersection of computer vision, NLP, and mathematical optimization by enabling automatic extraction and solving of optimization problems directly from images.

Abstract

This study presents AutoOpt-11k, a unique image dataset of over 11,000 handwritten and printed mathematical optimization models corresponding to single-objective, multi-objective, multi-level, and stochastic optimization problems exhibiting various types of complexities such as non-linearity, non-convexity, non-differentiability, discontinuity, and high-dimensionality. The labels consist of the LaTeX representation for all the images and modeling language representation for a subset of images. The dataset is created by 25 experts following ethical data creation guidelines and verified in two-phases to avoid errors. Further, we develop AutoOpt framework, a machine learning based automated approach for solving optimization problems, where the user just needs to provide an image of the formulation and AutoOpt solves it efficiently without any further human intervention. AutoOpt framework consists of three Modules: (i) M1 (Image_to_Text)- a deep learning model performs the Mathematical Expression Recognition (MER) task to generate the LaTeX code corresponding to the optimization formulation in image; (ii) M2 (Text_to_Text)- a small-scale fine-tuned LLM generates the PYOMO script (optimization modeling language) from LaTeX code; (iii) M3 (Optimization)- a Bilevel Optimization based Decomposition (BOBD) method solves the optimization formulation described in the PYOMO script. We use AutoOpt-11k dataset for training and testing of deep learning models employed in AutoOpt. The deep learning model for MER task (M1) outperforms ChatGPT, Gemini and Nougat on BLEU score metric. BOBD method (M3), which is a hybrid approach, yields better results on complex test problems compared to common approaches, like interior-point algorithm and genetic algorithm.
Paper Structure (13 sections, 14 equations, 11 figures, 11 tables, 1 algorithm)

This paper contains 13 sections, 14 equations, 11 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. AutoOpt-11k Dataset
  3. Framework for Automating Optimization Problem Solving
  4. Module M1: Image to LaTeX Code Generation
  5. Experimental Results
  6. Module M2: LaTeX to PYOMO Script Generation
  7. Module M3: Optimization using Bilevel Optimization based Decomposition Method
  8. Performance of AutoOpt Framework
  9. Conclusions
  10. Appendix: Dataset
  11. Appendix: Module M1
  12. Appendix: Module M2
  13. Appendix: Module M3

Figures (11)

  • Figure 1: AutoOpt framework: workflow demonstration using an example
  • Figure 2: A sample of 7 images from the AutoOpt-11k dataset.
  • Figure 3: Architecture of the deep learning model developed for MER
  • Figure 4: BLUE Score
  • Figure 5: Problem solving using Bilevel optimization-based decomposition approach
  • ...and 6 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2