Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems

Yuxiao Huang, Wenjie Zhang, Liang Feng, Xingyu Wu, Kay Chen Tan

TL;DR

The paper addresses the inadequacy of text-only prompts for high-dimensional optimization by introducing a multimodal LLM framework that processes both textual and visual prompts to better capture inter-variable dependencies, demonstrated on the capacitated vehicle routing problem (CVRP). The method employs a three-step workflow—heuristic extraction from solved problems, solution generation via learned heuristics, and solution evaluation/refinement—enabled by an XML-based representation and visual prompts. Empirical results show that the multimodal approach (MLLM-V) outperforms a text-only baseline (MLLM-T) across diverse CVRP benchmarks, delivering richer learned heuristics and improved solution quality. This work highlights the potential of multimodal prompts to extend LLM-based optimization to complex, real-world combinatorial problems and beyond.

Abstract

Recently, large language models (LLMs) have notably positioned them as capable tools for addressing complex optimization challenges. Despite this recognition, a predominant limitation of existing LLM-based optimization methods is their struggle to capture the relationships among decision variables when relying exclusively on numerical text prompts, especially in high-dimensional problems. Keeping this in mind, we first propose to enhance the optimization performance using multimodal LLM capable of processing both textual and visual prompts for deeper insights of the processed optimization problem. This integration allows for a more comprehensive understanding of optimization problems, akin to human cognitive processes. We have developed a multimodal LLM-based optimization framework that simulates human problem-solving workflows, thereby offering a more nuanced and effective analysis. The efficacy of this method is evaluated through extensive empirical studies focused on a well-known combinatorial optimization problem, i.e., capacitated vehicle routing problem. The results are compared against those obtained from the LLM-based optimization algorithms that rely solely on textual prompts, demonstrating the significant advantages of our multimodal approach.

How Multimodal Integration Boost the Performance of LLM for Optimization: Case Study on Capacitated Vehicle Routing Problems

TL;DR

The paper addresses the inadequacy of text-only prompts for high-dimensional optimization by introducing a multimodal LLM framework that processes both textual and visual prompts to better capture inter-variable dependencies, demonstrated on the capacitated vehicle routing problem (CVRP). The method employs a three-step workflow—heuristic extraction from solved problems, solution generation via learned heuristics, and solution evaluation/refinement—enabled by an XML-based representation and visual prompts. Empirical results show that the multimodal approach (MLLM-V) outperforms a text-only baseline (MLLM-T) across diverse CVRP benchmarks, delivering richer learned heuristics and improved solution quality. This work highlights the potential of multimodal prompts to extend LLM-based optimization to complex, real-world combinatorial problems and beyond.

Abstract

Recently, large language models (LLMs) have notably positioned them as capable tools for addressing complex optimization challenges. Despite this recognition, a predominant limitation of existing LLM-based optimization methods is their struggle to capture the relationships among decision variables when relying exclusively on numerical text prompts, especially in high-dimensional problems. Keeping this in mind, we first propose to enhance the optimization performance using multimodal LLM capable of processing both textual and visual prompts for deeper insights of the processed optimization problem. This integration allows for a more comprehensive understanding of optimization problems, akin to human cognitive processes. We have developed a multimodal LLM-based optimization framework that simulates human problem-solving workflows, thereby offering a more nuanced and effective analysis. The efficacy of this method is evaluated through extensive empirical studies focused on a well-known combinatorial optimization problem, i.e., capacitated vehicle routing problem. The results are compared against those obtained from the LLM-based optimization algorithms that rely solely on textual prompts, demonstrating the significant advantages of our multimodal approach.
Paper Structure (12 sections, 3 figures, 2 tables)

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Capacitated Vehicle Routing Problem
  4. Literature Review
  5. Multimodal LLM for Optimization
  6. Heuristic Extraction from Solved Problems
  7. Solution Generation via learned heuristics
  8. Solution Evaluation and Refinement
  9. Experimental Study
  10. Results
  11. Why MLLM-V performs better than MLLM-T
  12. Conclusion

Figures (3)

  • Figure 1: Illustration of a CVRP with optimal visiting routes.
  • Figure 2: Workflow of our proposed method with three steps.
  • Figure 3: Traveling routes obtained based on randomly generated, MLLM-T (i.e., "Text Prompt Only"), and MLLM-V (i.e., "With Vision Prompt"), respectively. "Optimal" shows the best routing solution of "P-n19-k2".