Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework

Yulong Hu, Tingting Dong, Sen Li

TL;DR

The paper tackles coordinating ride-pooling with public transit in multimodal networks by framing each ride-pooling vehicle as an MDP and proposing an offline training plus online fine-tuning RL framework. It introduces Reward-Guided Conservative Q-learning (RG-CQL), combining a Conservative Double Deep Q Network (CDDQN) with a supervised reward Guider to guide exploration during online updates. The method is validated on real Manhattan data, showing that pooling-transit coordination yields higher system rewards than standalone ride-pooling or solo rides with transit, and that offline training substantially boosts data efficiency with improved stability during online adaptation. Overall, RG-CQL mitigates the offline-online gap, reduces overestimation, and offers a scalable, data-efficient approach for large-scale multimodal ride-pooling systems integrated with transit, with meaningful implications for urban mobility and transit-oriented planning.

Abstract

This paper introduces a novel reinforcement learning (RL) framework, termed Reward-Guided Conservative Q-learning (RG-CQL), to enhance coordination between ride-pooling and public transit within a multimodal transportation network. We model each ride-pooling vehicle as an agent governed by a Markov Decision Process (MDP) and propose an offline training and online fine-tuning RL framework to learn the optimal operational decisions of the multimodal transportation systems, including rider-vehicle matching, selection of drop-off locations for passengers, and vehicle routing decisions, with improved data efficiency. During the offline training phase, we develop a Conservative Double Deep Q Network (CDDQN) as the action executor and a supervised learning-based reward estimator, termed the Guider Network, to extract valuable insights into action-reward relationships from data batches. In the online fine-tuning phase, the Guider Network serves as an exploration guide, aiding CDDQN in effectively and conservatively exploring unknown state-action pairs. The efficacy of our algorithm is demonstrated through a realistic case study using real-world data from Manhattan. We show that integrating ride-pooling with public transit outperforms two benchmark cases solo rides coordinated with transit and ride-pooling without transit coordination by 17% and 22% in the achieved system rewards, respectively. Furthermore, our innovative offline training and online fine-tuning framework offers a remarkable 81.3% improvement in data efficiency compared to traditional online RL methods with adequate exploration budgets, with a 4.3% increase in total rewards and a 5.6% reduction in overestimation errors. Experimental results further demonstrate that RG-CQL effectively addresses the challenges of transitioning from offline to online RL in large-scale ride-pooling systems integrated with transit.

Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework

TL;DR

The paper tackles coordinating ride-pooling with public transit in multimodal networks by framing each ride-pooling vehicle as an MDP and proposing an offline training plus online fine-tuning RL framework. It introduces Reward-Guided Conservative Q-learning (RG-CQL), combining a Conservative Double Deep Q Network (CDDQN) with a supervised reward Guider to guide exploration during online updates. The method is validated on real Manhattan data, showing that pooling-transit coordination yields higher system rewards than standalone ride-pooling or solo rides with transit, and that offline training substantially boosts data efficiency with improved stability during online adaptation. Overall, RG-CQL mitigates the offline-online gap, reduces overestimation, and offers a scalable, data-efficient approach for large-scale multimodal ride-pooling systems integrated with transit, with meaningful implications for urban mobility and transit-oriented planning.

Abstract

This paper introduces a novel reinforcement learning (RL) framework, termed Reward-Guided Conservative Q-learning (RG-CQL), to enhance coordination between ride-pooling and public transit within a multimodal transportation network. We model each ride-pooling vehicle as an agent governed by a Markov Decision Process (MDP) and propose an offline training and online fine-tuning RL framework to learn the optimal operational decisions of the multimodal transportation systems, including rider-vehicle matching, selection of drop-off locations for passengers, and vehicle routing decisions, with improved data efficiency. During the offline training phase, we develop a Conservative Double Deep Q Network (CDDQN) as the action executor and a supervised learning-based reward estimator, termed the Guider Network, to extract valuable insights into action-reward relationships from data batches. In the online fine-tuning phase, the Guider Network serves as an exploration guide, aiding CDDQN in effectively and conservatively exploring unknown state-action pairs. The efficacy of our algorithm is demonstrated through a realistic case study using real-world data from Manhattan. We show that integrating ride-pooling with public transit outperforms two benchmark cases solo rides coordinated with transit and ride-pooling without transit coordination by 17% and 22% in the achieved system rewards, respectively. Furthermore, our innovative offline training and online fine-tuning framework offers a remarkable 81.3% improvement in data efficiency compared to traditional online RL methods with adequate exploration budgets, with a 4.3% increase in total rewards and a 5.6% reduction in overestimation errors. Experimental results further demonstrate that RG-CQL effectively addresses the challenges of transitioning from offline to online RL in large-scale ride-pooling systems integrated with transit.
Paper Structure (24 sections, 18 equations, 12 figures, 8 tables, 1 algorithm)

This paper contains 24 sections, 18 equations, 12 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Order Matching for Ride-Pooling Platforms
  4. Coordinating Ride-hailing Services with Public Transit
  5. Offline RL and its Deployment in Ride-Sourcing Dispatch
  6. Recent Advances and Fine-tuning Dilemma in Offline RL
  7. Problem Description
  8. Model Formulation
  9. MDP Model
  10. Model Elements
  11. Policy and Platform Objective
  12. Real-time Order Dispatch and Drop-off Location Choices
  13. RG-CQL for Value Function and Policy Learning
  14. CDDQN for Offline Learning from Pre-collected Transitions
  15. Reward Guided CDDQN for Online-finetuning
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Problem setup with coordinated ride-pooling and transit services (Part of icons are from GoogleImages)
  • Figure 2: Intuitive example of MDP formulation. Consider at time $t$, agent $n$ just picked up rider 1 and observe matched rider 2, $l_{n,t}= 1, v_{n,t}= 2, o_{m}= 2, d_{m}= 5, p_{n,t}= (3,0,0,8,0,0,0,0,0)$, where the first element '3' specifies the destination zone of rider 1 and '8' denotes his remaining time onboard in minutes. The vehicle agent is assigned an action $a_{n,t} = 4$ to drop off the passenger at zone 4, allowing rider 2 to continue his/her journey via public transit. After the match, with the update of new route (in black arrows), rider 1 experiences 1 minute additional detour and remaining onboard time is updated to 9 minute. Rider 2 experiences 6 minute detour due to the combined usage of ride-pooling and transit and the remaining onboard time is 11 minutes. Consequently, the passenger information on board will update to $p_{n,t+1} = (3,4,0,9,11,0,1,6,0)$, where '3' and '4' denote the destination zones of riders 1 and 2, respectively; '9' and '11' represent the respective updated remaining times onboard; and '1' and '6' reflect the additional detours for riders 1 and 2, respectively. The remaining elements are set to zero, reflecting the vacancy of the third seat.
  • Figure 3: Visualization of bipartite matching
  • Figure 4: Comparison of naive CQL family pipeline (on the left) with RG-CQL (on the right) (part of icons from GoogleImages)
  • Figure 5: Overview of RG-CQL Framework
  • ...and 7 more figures

Theorems & Definitions (2)

  • Remark 1
  • Remark 2