Effective Reward Specification in Deep Reinforcement Learning

TL;DR

This work addresses the persistent challenge of reward specification in deep reinforcement learning, where improper design leads to misaligned or inefficient agents. It surveys reward composition and reward modeling, then presents four contributions: ASAF for imitation without policy optimization, coordination-promoting policy regularization in multi-agent RL, a constrained RL framework for direct behavior specification, and goal-conditioned GFlowNets for controllable multi-objective molecular design. Together, these approaches show how combining demonstrations, auxiliary objectives, explicit constraints, and goal-conditioned generation can improve sample efficiency, alignment, and controllability in complex tasks. The results illustrate both the potential and the limitations of each paradigm, emphasizing that no universal solution exists and that task-specific tool selection is essential. The work highlights practical implications for real-world RL deployment in robotics, multi-agent systems, and drug design, where intuitive interfaces for specifying objectives and constraints can reduce engineering effort and improve safety and performance.

Abstract

In the last decade, Deep Reinforcement Learning has evolved into a powerful tool for complex sequential decision-making problems. It combines deep learning's proficiency in processing rich input signals with reinforcement learning's adaptability across diverse control tasks. At its core, an RL agent seeks to maximize its cumulative reward, enabling AI algorithms to uncover novel solutions previously unknown to experts. However, this focus on reward maximization also introduces a significant difficulty: improper reward specification can result in unexpected, misaligned agent behavior and inefficient learning. The complexity of accurately specifying the reward function is further amplified by the sequential nature of the task, the sparsity of learning signals, and the multifaceted aspects of the desired behavior. In this thesis, we survey the literature on effective reward specification strategies, identify core challenges relating to each of these approaches, and propose original contributions addressing the issue of sample efficiency and alignment in deep reinforcement learning. Reward specification represents one of the most challenging aspects of applying reinforcement learning in real-world domains. Our work underscores the absence of a universal solution to this complex and nuanced challenge; solving it requires selecting the most appropriate tools for the specific requirements of each unique application.

Figures (42)

• Figure 1: Markov Chain over state-action pairs.
• Figure 1: Comparison between ASAF-1 and ASQF, our two transition-wise methods, on environments with increasing observation space dimensionality
• Figure 1: Hyperparameter tuning results for all algorithms. There is one distribution per (algorithm, environment) pair, each one formed of 50 data-points (hyperparameter configuration samples). Each point represents the best model performance averaged over 100 evaluation episodes and averaged over the 3 training seeds for one sampled hyperparameters configuration. The box-plots divide in quartiles the 49 lower-performing configurations for each distribution while the score of the best-performing configuration is highlighted above the box-plots by a single dot.
• Figure 1: Also see Figure \ref{['fig:reward_engineering']}. When enforcing 3 behavioral requirements with reward engineering, an ever larger proportion of the experiments are wasted finding either low-performing policies or policies that do not satisfy the behavioral constraints. In this case, none of the 343 experiments yielded a feasible policy that also solves the task (success rate near 1.0), showcasing that reward engineering scales poorly with the number of constraints due to the curse of dimensionality and to the composing effect of the multiple constraints in narrowing the space of feasible policies.
• Figure 1: Learned conditional-distributions for different focus regions passed as input to the same model. Each dot marks the image of a generated molecule in the objective space. The colors indicate how densely populated a particular area of the objective space is (brighter is denser). The focus regions (goal regions) are depicted in light blue. The distribution on the last row, second column, showcases a focus region which seems difficult to reach and may not contain as large a population of molecules in the state space. In such cases, the model cannot learn to consistently produce samples from that goal region when conditioned on this goal direction $d_g$ and will instead produce several samples very similar to the sampling distribution of an untrained model (uniform across the state space).

Theorems & Definitions (6)

• Lemma 1
• Theorem 1
• proof
• proof
• Theorem 2
• proof