Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Matching via Reinforcement Learning: An Expert Policy Orchestration Strategy

Chiara Mignacco, Matthieu Jonckheere, Gilles Stoltz

TL;DR

This work tackles online matching by learning to orchestrate a set of interpretable expert policies through advantage-based updates within an Adv2-inspired framework. It provides both expectation and high-probability regret guarantees and derives a novel finite-time bias bound for temporal-difference learning to enable reliable estimated advantages. A scalable neural actor-critic architecture implements the learned mixture over experts, ensuring real-time applicability in high-dimensional settings such as organ exchange networks. Empirical results on stochastic matching models demonstrate faster convergence and higher system efficiency than individual experts and standard RL baselines, highlighting the value of structured, adaptive learning for complex resource allocation tasks.

Abstract

Online matching problems arise in many complex systems, from cloud services and online marketplaces to organ exchange networks, where timely, principled decisions are critical for maintaining high system performance. Traditional heuristics in these settings are simple and interpretable but typically tailored to specific operating regimes, which can lead to inefficiencies when conditions change. We propose a reinforcement learning (RL) approach that learns to orchestrate a set of such expert policies, leveraging their complementary strengths in a data-driven, adaptive manner. Building on the Adv2 framework (Jonckheere et al., 2024), our method combines expert decisions through advantage-based weight updates and extends naturally to settings where only estimated value functions are available. We establish both expectation and high-probability regret guarantees and derive a novel finite-time bias bound for temporal-difference learning, enabling reliable advantage estimation even under constant step size and non-stationary dynamics. To support scalability, we introduce a neural actor-critic architecture that generalizes across large state spaces while preserving interpretability. Simulations on stochastic matching models, including an organ exchange scenario, show that the orchestrated policy converges faster and yields higher system level efficiency than both individual experts and conventional RL baselines. Our results highlight how structured, adaptive learning can improve the modeling and management of complex resource allocation and decision-making processes.

Online Matching via Reinforcement Learning: An Expert Policy Orchestration Strategy

TL;DR

This work tackles online matching by learning to orchestrate a set of interpretable expert policies through advantage-based updates within an Adv2-inspired framework. It provides both expectation and high-probability regret guarantees and derives a novel finite-time bias bound for temporal-difference learning to enable reliable estimated advantages. A scalable neural actor-critic architecture implements the learned mixture over experts, ensuring real-time applicability in high-dimensional settings such as organ exchange networks. Empirical results on stochastic matching models demonstrate faster convergence and higher system efficiency than individual experts and standard RL baselines, highlighting the value of structured, adaptive learning for complex resource allocation tasks.

Abstract

Online matching problems arise in many complex systems, from cloud services and online marketplaces to organ exchange networks, where timely, principled decisions are critical for maintaining high system performance. Traditional heuristics in these settings are simple and interpretable but typically tailored to specific operating regimes, which can lead to inefficiencies when conditions change. We propose a reinforcement learning (RL) approach that learns to orchestrate a set of such expert policies, leveraging their complementary strengths in a data-driven, adaptive manner. Building on the Adv2 framework (Jonckheere et al., 2024), our method combines expert decisions through advantage-based weight updates and extends naturally to settings where only estimated value functions are available. We establish both expectation and high-probability regret guarantees and derive a novel finite-time bias bound for temporal-difference learning, enabling reliable advantage estimation even under constant step size and non-stationary dynamics. To support scalability, we introduce a neural actor-critic architecture that generalizes across large state spaces while preserving interpretability. Simulations on stochastic matching models, including an organ exchange scenario, show that the orchestrated policy converges faster and yields higher system level efficiency than both individual experts and conventional RL baselines. Our results highlight how structured, adaptive learning can improve the modeling and management of complex resource allocation and decision-making processes.

Paper Structure

This paper contains 81 sections, 5 theorems, 66 equations, 9 figures, 13 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Theoretical Guarantees for Policy Orchestration with Learned Advantages
  3. A Scalable and Interpretable Framework for Real-Time Policy Learning
  4. Organization of the paper
  5. Orchestration of expert policies: setting and objectives
  6. Learning Strategies and Convergence Rates
  7. Potential-Based Strategies and Regret Guarantees
  8. Policy Orchestration based on Estimated Values
  9. Analysis in high probability
  10. Ensuring Assumption \ref{['lm:est-eps']} via Temporal Difference Learning
  11. Temporal-Difference Updates in Tabular Setting
  12. Finite-time bias bound for TD learning
  13. Corollary
  14. Remark
  15. Related Work and Comparison
  16. ...and 66 more sections

Key Result

Lemma 1

Suppose a learning strategy selects weights $\{q_t\}$ over $K$ experts, producing a sequence of policies $\{q_t\Pi\}$. For the strategies described above, the following regret bound holds. For any fixed policy $q\Pi \in \mathcal{C}(\Pi)$ and any $T > 1$, where $B_{T, K}$ can be made sublinear in $T$ using appropriate strategies.

Figures (9)

  • Figure 1: Neural network $\mathcal{N}_{\theta}$ with parameters $\theta \in \Theta$: two fully connected layers with ReLU activation, followed by a linear output layer.
  • Figure 4: Comparison of direct tabular learning performance against baseline methods.
  • Figure 5: Convergence of average performance under different orchestration and learning schemes. The legend (bottom right) is shared across all plots.
  • Figure 6: Learning curves for the donor exchange experiment.
  • Figure 7: Learning curves for the donor exchange experiment.
  • ...and 4 more figures

Theorems & Definitions (11)

  • Lemma 1: jonckheere2023symphonyexpertsorchestrationadversarial
  • Theorem 1: Control in Expectation
  • Theorem 2: high-probability control
  • Remark 1
  • Lemma 2: Bias contraction bound under stationary policy
  • Remark 2
  • Remark 3
  • Lemma 3: performance difference lemma; see KL02
  • proof : Proof of Theorem \ref{['th:cesaro']}
  • proof : proof of Theorem \ref{['th:HP-estim']}
  • ...and 1 more