Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Central Role of the Loss Function in Reinforcement Learning

Kaiwen Wang, Nathan Kallus, Wen Sun

TL;DR

It is proved that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss.

Abstract

This paper illustrates the central role of loss functions in data-driven decision making, providing a comprehensive survey on their influence in cost-sensitive classification (CSC) and reinforcement learning (RL). We demonstrate how different regression loss functions affect the sample efficiency and adaptivity of value-based decision making algorithms. Across multiple settings, we prove that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss. Moreover, we prove that distributional algorithms using the maximum likelihood loss achieve second-order bounds scaling with the policy variance and are even sharper than first-order bounds. This in particular proves the benefits of distributional RL. We hope that this paper serves as a guide analyzing decision making algorithms with varying loss functions, and can inspire the reader to seek out better loss functions to improve any decision making algorithm.

The Central Role of the Loss Function in Reinforcement Learning

TL;DR

It is proved that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss.

Abstract

This paper illustrates the central role of loss functions in data-driven decision making, providing a comprehensive survey on their influence in cost-sensitive classification (CSC) and reinforcement learning (RL). We demonstrate how different regression loss functions affect the sample efficiency and adaptivity of value-based decision making algorithms. Across multiple settings, we prove that algorithms using the binary cross-entropy loss achieve first-order bounds scaling with the optimal policy's cost and are much more efficient than the commonly used squared loss. Moreover, we prove that distributional algorithms using the maximum likelihood loss achieve second-order bounds scaling with the policy variance and are even sharper than first-order bounds. This in particular proves the benefits of distributional RL. We hope that this paper serves as a guide analyzing decision making algorithms with varying loss functions, and can inspire the reader to seek out better loss functions to improve any decision making algorithm.
Paper Structure (26 sections, 37 theorems, 111 equations, 2 tables, 7 algorithms)

This paper contains 26 sections, 37 theorems, 111 equations, 2 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Cost-Sensitive Classification
  3. Solving CSC with Squared-Loss Regression
  4. The Second-Order Lemma
  5. Regression with the Binary-Cross-Entropy Loss: First-Order PAC Bounds for CSC
  6. Maximum Likelihood Estimation: Second-Order PAC Bounds for CSC
  7. Improved Second-Order PAC Bounds for CSC with Pessimistic MLE
  8. Proof of the Second-Order Lemma
  9. Lower Bounds for CSC
  10. Plug-in Squared Loss Cannot Achieve First-Order
  11. Plug-in BCE Loss Cannot Achieve Second-Order
  12. Setup for Reinforcement Learning
  13. Problem Setup
  14. Online RL with Squared-Loss Regression
  15. Verifying Assumptions for Low-Rank MDPs
  16. ...and 11 more sections

Key Result

Theorem 1

Under ass:csc-realizability, for any $\delta\in(0,1)$, $1-\delta$, plug-in squared loss regression enjoys

Theorems & Definitions (62)

  • Theorem 1
  • Lemma 1: Second-Order Mean Comparison
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Theorem 2
  • Remark 1
  • Lemma 4
  • Theorem 3
  • ...and 52 more