Markov Decision Process and Approximate Dynamic Programming for a Patient Assignment Scheduling problem

TL;DR

This work models the Patient Assignment Scheduling problem in stochastic hospital environments as an infinite-horizon Markov Decision Process and seeks to minimize the long-run average cost. It introduces an Approximate Dynamic Programming approach to overcome the curse of dimensionality, employing basis-function features and approximate policy iteration to compute near-optimal policies from data-driven, large-scale hospital settings. The methodology is validated on both small, tractable and realistically-sized instances fitted to Australian hospital data, showing near-optimal performance and substantial reductions in transfers and nonprimary-ward occupancy. The findings provide a scalable, data-driven decision-support framework for hospital bed management with potential to improve patient flow and throughput in real-world settings.

Abstract

We study the Patient Assignment Scheduling (PAS) problem in a random environment that arises in the management of patient flow in the hospital systems, due to the stochastic nature of the arrivals as well as the Length of Stay distribution. We develop a Markov Decision Process (MDP) which aims to assign the newly arrived patients in an optimal way so as to minimise the total expected long-run cost per unit time over an infinite horizon. We assume Poisson arrival rates that depend on patient types, and Length of Stay distributions that depend on whether patients stay in their primary wards or not. Since the instances of realistic size of this problem are not easy to solve, we develop numerical methods based on Approximate Dynamic Programming. We illustrate the theory with numerical examples with parameters obtained by fitting to data from a tertiary referral hospital in Australia, and demonstrate the application potential of our methodology under practical considerations.

Figures (10)

• Figure 1: Decision epochs of the PAS problem. At the start of time period $d$ we observe some state $s$ and then make some decision $a$ about how to allocate/transfer patients. This transforms the system into a post decision state $s^{(a)}$ and then the system evolves in a stochastic manner, until new state $s'$ is observed at the start of the next time period $d+1$.
• Figure 2: Values of $\theta^{(f)}_n$ in iteration $n$ of Algorithm \ref{['Al1']} in in Example \ref{['ex1']}. Simulations were run for $M= 10^3, 10^4, 10^5$ and $10^6$ states, respectively.
• Figure 3: Values of $E_n$ (blue) compared to the optimal value $E^*=0.4098$ (red) in iteration $n$ of Algorithm \ref{['Al1']} in Example \ref{['ex1']}. Simulations were run for $M= 10^3, 10^4, 10^5$ and $10^6$ states, respectively.
• Figure 4: Simulation of a system with limited capacity under policy that applies $a=1$ (no transfers): $N^{(a)}_{k,i}(t)$ is the number of type $i$ patients in ward $k$, $\widehat{N}^{(a)}(t)=\sum_{k\not=i}N^{(a)}_{k,i}(t)$ is the number of patients in nonprimary wards on day $t$, post decision. We apply \ref{['eq:order']} and the parameters from Table \ref{['tab:parametersModel']}.
• Figure 5: Simulation of a system with limited capacity under policy that applies $a=2$ (with no more than $y=4$ transfers): We note the reduction of the total number of patients in nonprimary wards in comparison to the output for $a=1$ in Figure \ref{['ex2_bounded_a1']}.

Theorems & Definitions (2)

• Example 1
• Example 2