Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Learning for Balancing Short-Term and Long-Term Rewards

Peng Wu, Ziyu Shen, Feng Xie, Zhongyao Wang, Chunchen Liu, Yan Zeng

TL;DR

This paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing.

Abstract

Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.

Policy Learning for Balancing Short-Term and Long-Term Rewards

TL;DR

This paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing.

Abstract

Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.
Paper Structure (23 sections, 8 theorems, 97 equations, 3 figures, 9 tables)

This paper contains 23 sections, 8 theorems, 97 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Notation and Setup
  5. Formulation
  6. Optimal Policy and Challenges
  7. Optimal Policy
  8. Challenges
  9. Identifiability
  10. Policy Learning for Balancing Short-Term and Long-Term Rewards
  11. Estimation of Short-Term and Long-Term Rewards
  12. Learning the Optimal Policy
  13. Extension to Multi-Valued Treatments and Multiple Short-Term and Long-Term Rewards
  14. Experiments
  15. Experimental Setup
  16. ...and 8 more sections

Key Result

Lemma 4.1

The optimal policy where $\arg \max$ is over all possible policies.

Figures (3)

  • Figure 1: Comparison of Naive-S, Naive-Y and our method with different missing ratios of $Y$ on IHDP and JOBS, where the metric is the balanced reward.
  • Figure 2: Comparison of Naive-S, Naive-Y and ours on IHDP and JOBS, where the metric is the balanced reward.
  • Figure 3: Comparison of Naive-S, Naive-Y and our method with different time steps and other fixed missing ratios $\{0.2, 0.4\}$ on IHDP and JOBS.

Theorems & Definitions (14)

  • Lemma 4.1
  • Proposition 5.3: Identifiability of $\mathbb{V}(\pi; y)$
  • Theorem 6.1: Efficiency Bounds of $\mathbb{V}(\pi; s)$ and $\mathbb{V}(\pi; y)$
  • Proposition 6.2
  • Proposition 6.3: Unbiasedness
  • Theorem 6.4: Asymptotic Properties
  • Proposition 6.5: Regret and Estimation Error
  • proof : Proof of Proposition \ref{['prop5-3']}
  • proof : Proof of Theorem \ref{['thm-EIF']}
  • proof : Proof of Proposition \ref{['prop-efficiency']}
  • ...and 4 more