Comparative Analysis of Discrete and Continuous Action Spaces in Reservoir Management and Inventory Control Problems

TL;DR

This work addresses exact solutions for stochastic decision problems with mixed discrete and continuous state/action spaces (DA-HMDPs and CA-HMDPs) in reservoir management and inventory control. It introduces a symbolic dynamic programming framework that uses case calculus and Extended ADDs (XADDs) to represent and manipulate piecewise polynomial value functions and rewards, removing restrictive assumptions of rectangular piecewise structures. The methodology supports continuous-action maximization and delta-function integration within SDP, and is demonstrated through Mars Rover, reservoir management, and multi-item inventory control experiments, revealing practical trade-offs and scalability considerations. The results show that SDP with XADDs can yield exact solutions for complex, non-rectangular, piecewise problems, while guiding future work on heuristic integration and improved approximation techniques for larger-scale systems.

Abstract

This paper presents a comparative analysis of discrete and continuous action spaces within the contexts of reservoir management and inventory control problems. We explore the computational trade-offs between discrete action discretizations and continuous action settings, focusing on their effects on time complexity and space requirements across different horizons. Our analysis includes a detailed evaluation of discretization levels in reservoir management, highlighting that finer discretizations approach the performance of continuous actions but at increased computational costs. For inventory control, we investigate deterministic and stochastic demand scenarios, demonstrating the exponential growth in time and space with increasing discrete actions and inventory items. We also introduce a novel symbolic approach for solving continuous problems in hybrid MDPs (H-MDPs), utilizing a new XADD data structure to manage piecewise symbolic value functions. Our results underscore the challenges of scaling solutions and provide insights into efficient handling of discrete and continuous action spaces in complex decision problems. Future research directions include exploring heuristic search methods and improved approximations for enhancing the practicality of exact solutions.

Figures (26)

• Figure 1: The ideal worth capability $V^2(x)$ for the CAIC issue is addressed by a XADD. To assess $V^2(x)$, follow the choice tree to a leaf, where the non-coincidental enunciation gives the worth, and the coincidental verbalization gives the ideal strategy $a = \pi^{*,2}(x)$. The right diagram shows the superior arrangement $\pi^2$, lined up with Scarf's answer.
• Figure 2: Network geography between state factors in the 2-thing consistent activity Inventory Control (CAIC) issue (Left); Unique bayes organization (DBN) structure addressing the change and award capability (Center); progress probabilities and prize capability as far as CPF and PLE for $x_1$ (Right).
• Figure 3: VI(HMDP, $H$) $\longrightarrow$$(V^h,\pi^{*,h})$
• Figure 4: Regress($V,a,\vec{y}$) $\longrightarrow$$Q$
• Figure 9: Comparison of the three decision diagrams: Binary decision diagrams (BDDs) with boolean leaves and decisions (Left) representing as shown in the truth table; Algebraic decision diagrams (ADDs) with boolean decision nodes and real values at the leaves (Middle) represented by the truth table;