Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Staff: Offline Reinforcement Learning and Fine-Tuned LLMs for Warehouse Staffing Optimization

Kalle Kujanpää, Yuying Zhu, Kristina Klinkner, Shervin Malmasi

Abstract

We investigate machine learning approaches for optimizing real-time staffing decisions in semi-automated warehouse sortation systems. Operational decision-making can be supported at different levels of abstraction, with different trade-offs. We evaluate two approaches, each in a matching simulation environment. First, we train custom Transformer-based policies using offline reinforcement learning on detailed historical state representations, achieving a 2.4% throughput improvement over historical baselines in learned simulators. In high-volume warehouse operations, improvements of this size translate to significant savings. Second, we explore LLMs operating on abstracted, human-readable state descriptions. These are a natural fit for decisions that warehouse managers make using high-level operational summaries. We systematically compare prompting techniques, automatic prompt optimization, and fine-tuning strategies. While prompting alone proves insufficient, supervised fine-tuning combined with Direct Preference Optimization on simulator-generated preferences achieves performance that matches or slightly exceeds historical baselines in a hand-crafted simulator. Our findings demonstrate that both approaches offer viable paths toward AI-assisted operational decision-making. Offline RL excels with task-specific architectures. LLMs support human-readable inputs and can be combined with an iterative feedback loop that can incorporate manager preferences.

Learning to Staff: Offline Reinforcement Learning and Fine-Tuned LLMs for Warehouse Staffing Optimization

Abstract

We investigate machine learning approaches for optimizing real-time staffing decisions in semi-automated warehouse sortation systems. Operational decision-making can be supported at different levels of abstraction, with different trade-offs. We evaluate two approaches, each in a matching simulation environment. First, we train custom Transformer-based policies using offline reinforcement learning on detailed historical state representations, achieving a 2.4% throughput improvement over historical baselines in learned simulators. In high-volume warehouse operations, improvements of this size translate to significant savings. Second, we explore LLMs operating on abstracted, human-readable state descriptions. These are a natural fit for decisions that warehouse managers make using high-level operational summaries. We systematically compare prompting techniques, automatic prompt optimization, and fine-tuning strategies. While prompting alone proves insufficient, supervised fine-tuning combined with Direct Preference Optimization on simulator-generated preferences achieves performance that matches or slightly exceeds historical baselines in a hand-crafted simulator. Our findings demonstrate that both approaches offer viable paths toward AI-assisted operational decision-making. Offline RL excels with task-specific architectures. LLMs support human-readable inputs and can be combined with an iterative feedback loop that can incorporate manager preferences.

Paper Structure

This paper contains 8 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Domain Description
  4. Decision Problem
  5. Evaluation Approach
  6. Offline Reinforcement Learning
  7. Large Language Models
  8. Discussion

Figures (3)

  • Figure 1: Predicted versus ground-truth output rates for the hand-crafted simulator. The data points cluster along the identity line (in red), which indicates that the simulator has been well-calibrated with no systematic over- or underprediction. $R^2 = 0.976$ confirms that the simulator captures almost all of the variance in the ground-truth output.
  • Figure 2: The TGNN architecture consists of repeated masked residual blocks. Each residual block integrates information along the stages, then between the lines, and finally across the time dimension.
  • Figure 3: The learning curve for Optimization by PROmpting (OPRO). We see that the performance improves steadily before convergence.