Towards Enabling Learning for Time-Varying finite horizon Sequential Decision-Making Problems*

TL;DR

A novel approach is proposed that reinterprets finite-horizon, time-varying Para-SDM problems as equivalent time-invariant problems through topography lifting and proves scalable for time-invariant problems by leveraging deep neural networks to learn optimal stage-invariant state-action value functions, enabling handling of large-scale scenarios.

Abstract

Parameterized Sequential Decision Making (Para-SDM) framework models a wide array of network design applications spanning supply-chain, transportation, and sensor networks. These problems entail sequential multi-stage optimization characterized by states, control actions, and cost functions dependent on designable parameters. The challenge is to determine both the sequential decision policy and parameters simultaneously to minimize cumulative stagewise costs. Many Para-SDM problems are NP-hard and often necessitate time-varying policies. Existing algorithms tackling finite-horizon time-varying Para-SDM problems struggle with scalability when faced with a large number of states. Conversely, the sole algorithm addressing infinite-horizon Para-SDM assumes time (stage)-invariance, yielding stationary policies. However, this approach proves scalable for time-invariant problems by leveraging deep neural networks to learn optimal stage-invariant state-action value functions, enabling handling of large-scale scenarios. This article proposes a novel approach that reinterprets finite-horizon, time-varying Para-SDM problems as equivalent time-invariant problems through topography lifting. Our method achieves nearly identical results to the time-varying solution while exhibiting improved performance times in various simulations, notably in the small cell network problem. This fresh perspective on Para-SDM problems expands the scope of addressable issues and holds promise for future scalability through the integration of learning methods.

Figures (5)

• Figure 1: Facility-Location Path-Optimization Problem in UAV transport network: (a) This problem shows $N=4$ UAVs $V_i$ each with different amount of initial battery charge values; and their target destinations are $\delta_i$. The objective is to determine the locations $y_j$ of charging UGV facilities $f_j$ as well as the routes (sequence of UGVs) of each UAV such that the cumulative travel-distances of all UAVs to their respective destinations is minimized. Note that UAV's have to hop onto UGVs for charging before if they have insufficient charge to reach their destination. (b) A stage-wise depiction of the FLPO problem. Here the FLPO path is reinterpreted in this graphical network, Here the set of possible UAV trajectories is divided into stages, where states in each stage comprise the facilities and the destination. A path comprises a sequence of states that a particular UAV takes. The objective is to determine the shortest path, while also determining the parameter associated (location of the facility) with each state.
• Figure 2: Stagewise FLPO Architecture. Green nodes demonstrate a transportation path from node $n_1$ to $\delta$ via stages $\left\{\Gamma_k\right\}_{k=1}^{M+1}$
• Figure 3: $x-y$ coordinates of the datasets generated for 5G small-cell simulations shown in 2D plane. Each figure depicts distributed user nodes as black points and the destination center as a solid blue triangle. Additionally, all coordinates are normalized to fit within a unit square, facilitating efficient hyperparameter tuning.
• Figure 4: Comparison of the solutions for the dataset 2, where optimal location of the facilities (Cell Towers) and the routes of user nodes are shown.
• Figure 5: The normalized cost and performance time comparison of the base time-invariant FLPO solution and the proposed time-invariant FLPO solution.