Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Review of the Deep Sea Treasure problem as a Multi-Objective Reinforcement Learning Benchmark

Amber Cassimon, Reinout Eyckerman, Siegfried Mercelis, Steven Latré, Peter Hellinckx

TL;DR

The paper reevaluates the Deep Sea Treasure benchmark as a multi-objective reinforcement learning testbed, showing the original bi-objective DST is relatively simple and may poorly reflect practical MORL problems. It then proposes an enhanced tri-objective variant with a substantially larger state-action space and a fuel objective, shifting the formulation toward SER-like criteria and highlighting new exploration-exploitation trade-offs. A reference Python implementation with gym compatibility, extensive configurability, and a publicly available Pareto front is provided to support reproducible SER-based benchmarking. The authors discuss constraints, SER-oriented research directions, and the continued status of DST as a toy benchmark that remains valuable for method development and controlled analysis in MORL.

Abstract

In this paper, the authors investigate the Deep Sea Treasure (DST) problem as proposed by Vamplew et al. Through a number of proofs, the authors show the original DST problem to be quite basic, and not always representative of practical Multi-Objective Optimization problems. In an attempt to bring theory closer to practice, the authors propose an alternative, improved version of the DST problem, and prove that some of the properties that simplify the original DST problem no longer hold. The authors also provide a reference implementation and perform a comparison between their implementation, and other existing open-source implementations of the problem. Finally, the authors also provide a complete Pareto-front for their new DST problem.

A Review of the Deep Sea Treasure problem as a Multi-Objective Reinforcement Learning Benchmark

TL;DR

The paper reevaluates the Deep Sea Treasure benchmark as a multi-objective reinforcement learning testbed, showing the original bi-objective DST is relatively simple and may poorly reflect practical MORL problems. It then proposes an enhanced tri-objective variant with a substantially larger state-action space and a fuel objective, shifting the formulation toward SER-like criteria and highlighting new exploration-exploitation trade-offs. A reference Python implementation with gym compatibility, extensive configurability, and a publicly available Pareto front is provided to support reproducible SER-based benchmarking. The authors discuss constraints, SER-oriented research directions, and the continued status of DST as a toy benchmark that remains valuable for method development and controlled analysis in MORL.

Abstract

In this paper, the authors investigate the Deep Sea Treasure (DST) problem as proposed by Vamplew et al. Through a number of proofs, the authors show the original DST problem to be quite basic, and not always representative of practical Multi-Objective Optimization problems. In an attempt to bring theory closer to practice, the authors propose an alternative, improved version of the DST problem, and prove that some of the properties that simplify the original DST problem no longer hold. The authors also provide a reference implementation and perform a comparison between their implementation, and other existing open-source implementations of the problem. Finally, the authors also provide a complete Pareto-front for their new DST problem.

Paper Structure

This paper contains 24 sections, 20 equations, 3 figures.

Table of Contents

  1. INTRODUCTION
  2. MORL
  3. MaOO
  4. MDP
  5. STATE OF THE ART
  6. Optimality Criteria
  7. Existing Benchmark Suites
  8. DST
  9. Open Source Implementations
  10. BI-OBJECTIVE DEEP SEA TREASURE
  11. State Space
  12. Action Space
  13. Dynamics
  14. Rewards
  15. TRI-OBJECTIVE DEEP SEA TREASURE
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Submarine moving through DST environment
  • Figure 2: 3-Objective Pareto-front. Points in blue lie on the convex hull, while points in red are contained inside the convex hull.
  • Figure 3: Detailed view of the low-treasure region of the 3-Objective Pareto-front. Points in blue lie on the convex hull, while points in red are contained inside the convex hull. A convex hull is the smallest, convex set of points that contains a given set of points.