Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning LLMs with Graph Neural Solvers for Combinatorial Optimization

Shaodi Feng, Zhuoyi Lin, Yaoxin Wu, Haiyan Yin, Yan Jin, Senthilnath Jayavelu, Xun Xu

Abstract

Recent research has demonstrated the effectiveness of large language models (LLMs) in solving combinatorial optimization problems (COPs) by representing tasks and instances in natural language. However, purely language-based approaches struggle to accurately capture complex relational structures inherent in many COPs, rendering them less effective at addressing medium-sized or larger instances. To address these limitations, we propose AlignOPT, a novel approach that aligns LLMs with graph neural solvers to learn a more generalizable neural COP heuristic. Specifically, AlignOPT leverages the semantic understanding capabilities of LLMs to encode textual descriptions of COPs and their instances, while concurrently exploiting graph neural solvers to explicitly model the underlying graph structures of COP instances. Our approach facilitates a robust integration and alignment between linguistic semantics and structural representations, enabling more accurate and scalable COP solutions. Experimental results demonstrate that AlignOPT achieves state-of-the-art results across diverse COPs, underscoring its effectiveness in aligning semantic and structural representations. In particular, AlignOPT demonstrates strong generalization, effectively extending to previously unseen COP instances.

Aligning LLMs with Graph Neural Solvers for Combinatorial Optimization

Abstract

Recent research has demonstrated the effectiveness of large language models (LLMs) in solving combinatorial optimization problems (COPs) by representing tasks and instances in natural language. However, purely language-based approaches struggle to accurately capture complex relational structures inherent in many COPs, rendering them less effective at addressing medium-sized or larger instances. To address these limitations, we propose AlignOPT, a novel approach that aligns LLMs with graph neural solvers to learn a more generalizable neural COP heuristic. Specifically, AlignOPT leverages the semantic understanding capabilities of LLMs to encode textual descriptions of COPs and their instances, while concurrently exploiting graph neural solvers to explicitly model the underlying graph structures of COP instances. Our approach facilitates a robust integration and alignment between linguistic semantics and structural representations, enabling more accurate and scalable COP solutions. Experimental results demonstrate that AlignOPT achieves state-of-the-art results across diverse COPs, underscoring its effectiveness in aligning semantic and structural representations. In particular, AlignOPT demonstrates strong generalization, effectively extending to previously unseen COP instances.

Paper Structure

This paper contains 31 sections, 11 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Combinatorial Optimization
  4. LLM for Combinatorial Optimization
  5. Preliminaries
  6. Combinatorial Optimization Problems
  7. Neural Construction Heuristics for COPs
  8. The Proposed Framework
  9. COP-Specific Text-Attributed Representations
  10. Graph-based Neural Encoder
  11. Aligning LLM with Graph-based Neural Solvers
  12. Text-Graph Contrastive (TGC) Loss
  13. Text-Graph Matching (TGM) Loss
  14. Fine-tuning Schemes
  15. Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Overall workflow of AlignOPT. (a) AlignOPT first performs multi-task pretraining on diverse COPs to align semantic and structural node representations with TGC and TGM losses. The LLM remains frozen and processes the TAIs to generate semantic node representations. (b) The encoder and decoder are then fine-tuned through reinforcement learning to solve COPs. Notably, LLMs are excluded during this phase to ensure computational efficiency, as the encoder has already been aligned with LLM-derived representations during pre-training. (c) The model architecture of the graph-based encoder, which applies a mixed attention mechanism that enables handling COPs represented by graphs.
  • Figure 2: Generalization results on 3 unseen COPs.
  • Figure 3: Average Objective values of different LLMs (Llama3.1 8B and Qwen2.5 8B)