Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybridising Reinforcement Learning and Heuristics for Hierarchical Directed Arc Routing Problems

Van Quang Nguyen, Quoc Chuong Nguyen, Thu Huong Dang, Truong-Son Hy

TL;DR

The paper tackles Hierarchical Directed Capacitated Arc Routing Problems ($HDCARP$) with two variants, $HDCARP$-U and $HDCARP$-P, both NP-hard and challenging for large-scale instances. It introduces HRDA, a Hybrid Reinforcement Learning and Heuristic Algorithm that combines an Adjacency Matrix Attention Encoder (AMAE) with a Transformer-like decoder and PPO-based training to dynamically guide local search. The approach achieves significant speedups over purely matheuristic or RL methods while preserving solution quality, demonstrated through extensive experiments on artificial HDCARP instances and supported by public code. The work advances scalable, robust routing for hierarchical arc-routing tasks and points to extensions to other ARP variants and multi-agent settings.

Abstract

The Hierarchical Directed Capacitated Arc Routing Problem (HDCARP) is an extension of the Capacitated Arc Routing Problem (CARP), where the arcs of a graph are divided into classes based on their priority. The traversal of these classes is determined by either precedence constraints or a hierarchical objective, resulting in two distinct HDCARP variants. To the best of our knowledge, only one matheuristic has been proposed for these variants, but it performs relatively slowly, particularly for large-scale instances (Ha et al., 2024). In this paper, we propose a fast heuristic to efficiently address the computational challenges of HDCARP. Furthermore, we incorporate Reinforcement Learning (RL) into our heuristic to effectively guide the selection of local search operators, resulting in a hybrid algorithm. We name this hybrid algorithm as the Hybrid Reinforcement Learning and Heuristic Algorithm for Directed Arc Routing (HRDA). The hybrid algorithm adapts to changes in the problem dynamically, using real-time feedback to improve routing strategies and solution's quality by integrating heuristic methods. Extensive computational experiments on artificial instances demonstrate that this hybrid approach significantly improves the speed of the heuristic without deteriorating the solution quality. Our source code is publicly available at: https://github.com/HySonLab/ArcRoute

Hybridising Reinforcement Learning and Heuristics for Hierarchical Directed Arc Routing Problems

TL;DR

The paper tackles Hierarchical Directed Capacitated Arc Routing Problems () with two variants, -U and -P, both NP-hard and challenging for large-scale instances. It introduces HRDA, a Hybrid Reinforcement Learning and Heuristic Algorithm that combines an Adjacency Matrix Attention Encoder (AMAE) with a Transformer-like decoder and PPO-based training to dynamically guide local search. The approach achieves significant speedups over purely matheuristic or RL methods while preserving solution quality, demonstrated through extensive experiments on artificial HDCARP instances and supported by public code. The work advances scalable, robust routing for hierarchical arc-routing tasks and points to extensions to other ARP variants and multi-agent settings.

Abstract

The Hierarchical Directed Capacitated Arc Routing Problem (HDCARP) is an extension of the Capacitated Arc Routing Problem (CARP), where the arcs of a graph are divided into classes based on their priority. The traversal of these classes is determined by either precedence constraints or a hierarchical objective, resulting in two distinct HDCARP variants. To the best of our knowledge, only one matheuristic has been proposed for these variants, but it performs relatively slowly, particularly for large-scale instances (Ha et al., 2024). In this paper, we propose a fast heuristic to efficiently address the computational challenges of HDCARP. Furthermore, we incorporate Reinforcement Learning (RL) into our heuristic to effectively guide the selection of local search operators, resulting in a hybrid algorithm. We name this hybrid algorithm as the Hybrid Reinforcement Learning and Heuristic Algorithm for Directed Arc Routing (HRDA). The hybrid algorithm adapts to changes in the problem dynamically, using real-time feedback to improve routing strategies and solution's quality by integrating heuristic methods. Extensive computational experiments on artificial instances demonstrate that this hybrid approach significantly improves the speed of the heuristic without deteriorating the solution quality. Our source code is publicly available at: https://github.com/HySonLab/ArcRoute
Paper Structure (38 sections, 15 equations, 2 figures, 6 tables, 9 algorithms)

This paper contains 38 sections, 15 equations, 2 figures, 6 tables, 9 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Capacitated arc routing problems
  4. ARPs with precedence relations
  5. Local search procedures
  6. Machine Learning for ARPs/VRP
  7. Heuristic algorithm
  8. Construction phase
  9. Local search phase
  10. Hybrid algorithm
  11. Encoder
  12. Decoder
  13. Training the policy network via reinforcement learning with heuristic integration
  14. Reinforcement Learning and Heuristic Integration
  15. Initial Solution Generation:
  16. ...and 23 more sections

Figures (2)

  • Figure 1: RL with Heuristic Integration.
  • Figure 2: Comparison of HRDA and RL-only methods