Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Non-Cumulative Objective

Wei Cui, Wei Yu

TL;DR

The paper addresses the limitation that many systems, especially in networking, have non-cumulative objectives that are not naturally expressed as reward sums. It introduces generalized Bellman updates, replacing the summation with a function $g(\cdot,\cdot)$ to optimize objectives like bottleneck, maximum, and harmonic mean, and proves convergence under clear conditions. Through theoretical results and multiple experiments (CartPole, Breakout, single-path maximum-flow routing, and wireless ad hoc routing), it demonstrates that generalized TD-style learning can achieve competitive or superior performance while enabling new problem formulations. The approach enables efficient learning with non-cumulative objectives in both single-agent and multi-agent settings, with practical impact on routing, spectrum access, and other control tasks where bottleneck or min-max criteria are central.

Abstract

In reinforcement learning, the objective is almost always defined as a \emph{cumulative} function over the rewards along the process. However, there are many optimal control and reinforcement learning problems in various application fields, especially in communications and networking, where the objectives are not naturally expressed as summations of the rewards. In this paper, we recognize the prevalence of non-cumulative objectives in various problems, and propose a modification to existing algorithms for optimizing such objectives. Specifically, we dive into the fundamental building block for many optimal control and reinforcement learning algorithms: the Bellman optimality equation. To optimize a non-cumulative objective, we replace the original summation operation in the Bellman update rule with a generalized operation corresponding to the objective. Furthermore, we provide sufficient conditions on the form of the generalized operation as well as assumptions on the Markov decision process under which the globally optimal convergence of the generalized Bellman updates can be guaranteed. We demonstrate the idea experimentally with the bottleneck objective, i.e., the objectives determined by the minimum reward along the process, on classical optimal control and reinforcement learning tasks, as well as on two network routing problems on maximizing the flow rates.

Reinforcement Learning with Non-Cumulative Objective

TL;DR

The paper addresses the limitation that many systems, especially in networking, have non-cumulative objectives that are not naturally expressed as reward sums. It introduces generalized Bellman updates, replacing the summation with a function to optimize objectives like bottleneck, maximum, and harmonic mean, and proves convergence under clear conditions. Through theoretical results and multiple experiments (CartPole, Breakout, single-path maximum-flow routing, and wireless ad hoc routing), it demonstrates that generalized TD-style learning can achieve competitive or superior performance while enabling new problem formulations. The approach enables efficient learning with non-cumulative objectives in both single-agent and multi-agent settings, with practical impact on routing, spectrum access, and other control tasks where bottleneck or min-max criteria are central.

Abstract

In reinforcement learning, the objective is almost always defined as a \emph{cumulative} function over the rewards along the process. However, there are many optimal control and reinforcement learning problems in various application fields, especially in communications and networking, where the objectives are not naturally expressed as summations of the rewards. In this paper, we recognize the prevalence of non-cumulative objectives in various problems, and propose a modification to existing algorithms for optimizing such objectives. Specifically, we dive into the fundamental building block for many optimal control and reinforcement learning algorithms: the Bellman optimality equation. To optimize a non-cumulative objective, we replace the original summation operation in the Bellman update rule with a generalized operation corresponding to the objective. Furthermore, we provide sufficient conditions on the form of the generalized operation as well as assumptions on the Markov decision process under which the globally optimal convergence of the generalized Bellman updates can be guaranteed. We demonstrate the idea experimentally with the bottleneck objective, i.e., the objectives determined by the minimum reward along the process, on classical optimal control and reinforcement learning tasks, as well as on two network routing problems on maximizing the flow rates.
Paper Structure (34 sections, 4 theorems, 38 equations, 6 figures, 3 tables)

This paper contains 34 sections, 4 theorems, 38 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Generalized Optimal Control & Reinforcement Learning Formulation
  3. Conventional Formulation
  4. Generalized Non-Cumulative Objectives
  5. Learning Algorithms with Generalized Bellman Updates
  6. Bellman Update with Generalized Operations
  7. Conditions for Convergence
  8. Suboptimality with Stochastic Transitions and Rewards
  9. Examples of Generalized Objectives and Bellman Update Operations
  10. Bottleneck Reward Objective
  11. Maximum Reward Objective
  12. Harmonic Mean Reward Objective
  13. Applications of Generalized Reinforcement Learning
  14. Classical Reinforcement Learning Problems with Bottleneck Objectives
  15. Single-Path Maximum-Flow Routing with Bottleneck Objective on a Graph
  16. ...and 19 more sections

Key Result

Theorem 1

On a single-agent fully observable MDP, the series of value functions obtained from iteratively applying the generalized Bellman update rule $Q\leftarrow F^*Q$ as in equ:bellman_update_modified_mapping1 is guaranteed to converge to a unique convergence point in $\mathcal{R}^{|\mathcal{V}|}$ from any

Figures (6)

  • Figure 1: Learning progress on CartPole: From 6 full training trails under 6 random seeds, we plot the mean as the line and the standard deviation as the shadow.
  • Figure 2: Learning progress on Breakout: From 3 full training trails under 3 random seeds, we plot the mean as the line and the standard deviation as the shadow.
  • Figure 3: The maximum network flow problem as modeled by a directed graph.
  • Figure 4: Learning progress of various algorithms on the physical layer routing problem: The dotted lines are the performance values of all update steps; the solid lines are the smoothed curves, with each point being the average over the last 10 update steps.
  • Figure 5: Cumulative distribution function of flow bottleneck rates in 1000 testing wireless ad hoc networks.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • proof
  • proof
  • proof
  • proof