Reward Engineering for Reinforcement Learning in Software Tasks

TL;DR

The paper presents the first systematic survey of reward and environment design for reinforcement learning in software engineering, arguing that practical SE tasks demand proxy and hybrid rewards rather than single numeric objectives. It offers a taxonomy based on reward source (execution, similarity, preference), granularity (token to trajectory), and aggregation (single, weighted, or learned mixing), and maps this taxonomy onto a broad set of code-centric tasks from generation to repair and testing. Key contributions include synthesizing design patterns (verifiable outcomes, hybrid signals, and granularity choices), highlighting how environment and reward interact, and providing recommendations to address misalignment, sparsity, and comparability challenges. The findings have practical impact by guiding practitioners to select robust reward formulations and by identifying avenues for more stable, multi-objective optimization in SE RL systems.

Abstract

Reinforcement learning is increasingly used for code-centric tasks. These tasks include code generation, summarization, understanding, repair, testing, and optimization. This trend is growing faster with large language models and autonomous agents. A key challenge is how to design reward signals that make sense for software. In many RL problems, the reward is a clear number. In software, this is often not possible. The goal is rarely a single numeric objective. Instead, rewards are usually proxies. Common proxies check if the code compiles, passes tests, or satisfies quality metrics. Many reward designs have been proposed for code-related tasks. However, the work is scattered across areas and papers. There is no single survey that brings these approaches together and shows the full landscape of reward design for RL in software. In this survey, we provide the first systematic and comprehensive review of reward engineering for RL in software tasks. We focus on existing methods and techniques. We structure the literature along three complementary dimensions, summarizing the reward-design choices within each. We conclude with challenges and recommendations in the reward design space for SE tasks.