Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for Efficient Returns Management

Pascal Linden, Nathalie Paul, Tim Wirtz, Stefan Wrobel

TL;DR

This work tackles online returns allocation in retail warehouses by casting it as an online multiple knapsack problem with a limited intermediate buffer. It introduces PostAlloc, a reinforcement learning framework with a policy network and a baseline, trained via REINFORCE, to decide among accept, reject, or postpone actions as items arrive. The method extends prior single-knapsack RL approaches to multiple knapsacks and a postponement mechanism, achieving near-offline optimality (gaps around 3%) while dramatically reducing average storage time (up to 100% in some setups; 96.1% for uncorrelated data). The results demonstrate substantial practical impact for returns management, enabling on-the-fly decisions with far lower storage costs, and point to future work on scaling to more stores and richer product representations.

Abstract

In retail warehouses, returned products are typically placed in an intermediate storage until a decision regarding further shipment to stores is made. The longer products are held in storage, the higher the inefficiency and costs of the returns management process, since enough storage area has to be provided and maintained while the products are not placed for sale. To reduce the average product storage time, we consider an alternative solution where reallocation decisions for products can be made instantly upon their arrival in the warehouse allowing only a limited number of products to still be stored simultaneously. We transfer the problem to an online multiple knapsack problem and propose a novel reinforcement learning approach to pack the items (products) into the knapsacks (stores) such that the overall value (expected revenue) is maximized. Empirical evaluations on simulated data demonstrate that, compared to the usual offline decision procedure, our approach comes with a performance gap of only 3% while significantly reducing the average storage time of a product by 96%.

Reinforcement Learning for Efficient Returns Management

TL;DR

This work tackles online returns allocation in retail warehouses by casting it as an online multiple knapsack problem with a limited intermediate buffer. It introduces PostAlloc, a reinforcement learning framework with a policy network and a baseline, trained via REINFORCE, to decide among accept, reject, or postpone actions as items arrive. The method extends prior single-knapsack RL approaches to multiple knapsacks and a postponement mechanism, achieving near-offline optimality (gaps around 3%) while dramatically reducing average storage time (up to 100% in some setups; 96.1% for uncorrelated data). The results demonstrate substantial practical impact for returns management, enabling on-the-fly decisions with far lower storage costs, and point to future work on scaling to more stores and richer product representations.

Abstract

In retail warehouses, returned products are typically placed in an intermediate storage until a decision regarding further shipment to stores is made. The longer products are held in storage, the higher the inefficiency and costs of the returns management process, since enough storage area has to be provided and maintained while the products are not placed for sale. To reduce the average product storage time, we consider an alternative solution where reallocation decisions for products can be made instantly upon their arrival in the warehouse allowing only a limited number of products to still be stored simultaneously. We transfer the problem to an online multiple knapsack problem and propose a novel reinforcement learning approach to pack the items (products) into the knapsacks (stores) such that the overall value (expected revenue) is maximized. Empirical evaluations on simulated data demonstrate that, compared to the usual offline decision procedure, our approach comes with a performance gap of only 3% while significantly reducing the average storage time of a product by 96%.
Paper Structure (27 sections, 1 equation, 2 figures, 3 tables)

This paper contains 27 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Problem Statement
  2. Motivating Example
  3. Translation to the Knapsack Problem
  4. Related Work
  5. Methods
  6. Reinforcement Learning Formalization
  7. State
  8. Action
  9. Reward
  10. Model Overview
  11. Policy
  12. Baseline network
  13. Empirical Evaluation
  14. Data generation
  15. Benchmarks
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Model architecture of PostAlloc. At each step $i$, the input state $s_i$ is fed into the policy and the baseline network. The policy network outputs a probability distribution over the actions accept, reject, and postpone while the output of the baseline network is used in both networks' gradient-based parameter updates.
  • Figure 2: Final accept and reject decisions of the network on all datasets with $K=3$ (top) and $K=7$ (bottom). Every item must be accepted or rejected independent of whether the decision has been postponed previously. The inner circle depicts the network decisions, while the accept decisions of the network are refined on the outer layer into the decisions of the environment to accept or reject the items. The environment always enforces the capacity constraints to be fulfilled. Thus, all cases marked with Env-Reject are attempts of the network to accept an item even though it does not fit into the current knapsack anymore. For $K=3$ (top), a well-balanced distribution of accept and reject decision has been learned for all datasets. For $K=7$ (bottom) this is only the case for the uncorrelated dataset. For those with correlation, the networks' policy converged into a take-all strategy meaning that no useful behavior was learned.