Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination

Haoran Su, Yandong Sun, Congjia Yu

TL;DR

This paper argues that hand-crafted reward engineering in multi-agent reinforcement learning is increasingly infeasible due to credit assignment, non-stationarity, and combinatorial complexity. It proposes a paradigm shift to language-based objectives enabled by large language models, organized around three pillars: semantic reward specification, dynamic adaptation, and human alignment by default. Supporting evidence from EUREKA, CARD, and RLVR demonstrates the potential of language-mediated reward generation and verifiable supervision to guide coordination, with pathways for offline reward generation and runtime language-based coordination. The work outlines a concrete experimental agenda to validate the approach, addresses key challenges such as computational cost, safety, ambiguity, and scalability, and sketches a future where semantic coordination replaces explicit numerical signals for scalable multi-agent coordination.

Abstract

Reward engineering, the manual specification of reward functions to induce desired agent behavior, remains a fundamental challenge in multi-agent reinforcement learning. This difficulty is amplified by credit assignment ambiguity, environmental non-stationarity, and the combinatorial growth of interaction complexity. We argue that recent advances in large language models (LLMs) point toward a shift from hand-crafted numerical rewards to language-based objective specifications. Prior work has shown that LLMs can synthesize reward functions directly from natural language descriptions (e.g., EUREKA) and adapt reward formulations online with minimal human intervention (e.g., CARD). In parallel, the emerging paradigm of Reinforcement Learning from Verifiable Rewards (RLVR) provides empirical evidence that language-mediated supervision can serve as a viable alternative to traditional reward engineering. We conceptualize this transition along three dimensions: semantic reward specification, dynamic reward adaptation, and improved alignment with human intent, while noting open challenges related to computational overhead, robustness to hallucination, and scalability to large multi-agent systems. We conclude by outlining a research direction in which coordination arises from shared semantic representations rather than explicitly engineered numerical signals.

The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination

TL;DR

This paper argues that hand-crafted reward engineering in multi-agent reinforcement learning is increasingly infeasible due to credit assignment, non-stationarity, and combinatorial complexity. It proposes a paradigm shift to language-based objectives enabled by large language models, organized around three pillars: semantic reward specification, dynamic adaptation, and human alignment by default. Supporting evidence from EUREKA, CARD, and RLVR demonstrates the potential of language-mediated reward generation and verifiable supervision to guide coordination, with pathways for offline reward generation and runtime language-based coordination. The work outlines a concrete experimental agenda to validate the approach, addresses key challenges such as computational cost, safety, ambiguity, and scalability, and sketches a future where semantic coordination replaces explicit numerical signals for scalable multi-agent coordination.

Abstract

Reward engineering, the manual specification of reward functions to induce desired agent behavior, remains a fundamental challenge in multi-agent reinforcement learning. This difficulty is amplified by credit assignment ambiguity, environmental non-stationarity, and the combinatorial growth of interaction complexity. We argue that recent advances in large language models (LLMs) point toward a shift from hand-crafted numerical rewards to language-based objective specifications. Prior work has shown that LLMs can synthesize reward functions directly from natural language descriptions (e.g., EUREKA) and adapt reward formulations online with minimal human intervention (e.g., CARD). In parallel, the emerging paradigm of Reinforcement Learning from Verifiable Rewards (RLVR) provides empirical evidence that language-mediated supervision can serve as a viable alternative to traditional reward engineering. We conceptualize this transition along three dimensions: semantic reward specification, dynamic reward adaptation, and improved alignment with human intent, while noting open challenges related to computational overhead, robustness to hallucination, and scalability to large multi-agent systems. We conclude by outlining a research direction in which coordination arises from shared semantic representations rather than explicitly engineered numerical signals.
Paper Structure (35 sections, 3 figures)

This paper contains 35 sections, 3 figures.

Table of Contents

  1. Introduction
  2. The multi-agent reward problem
  3. Our thesis
  4. Evidence and scope
  5. Background: The broken promise of reward engineering
  6. Reward design in single-agent RL
  7. Why multi-agent makes it harder
  8. Current approaches and their limitations
  9. The LLM opportunity
  10. What LLMs bring to the table
  11. LLMs for reward generation
  12. From generation to coordination
  13. Three pillars of language-based objectives
  14. Pillar 1: Semantic reward specification
  15. Pillar 2: Dynamic adaptation
  16. ...and 20 more sections

Figures (3)

  • Figure 1: The paradigm shift from reward engineering to language-based objectives. (Left) Traditional approach requires human engineers to manually design reward functions through iterative trial-and-error. (Right) LLM-based approach allows humans to specify objectives in natural language, with the LLM generating and refining reward functions automatically based on behavioral feedback.
  • Figure 2: Two distinct pathways for LLM-enabled multi-agent coordination. Pathway 1 (left) uses LLMs to generate reward functions offline; agents then train via standard MARL without LLM involvement at runtime. Pathway 2 (right) embeds LLMs directly as agent controllers, enabling natural language coordination at runtime. These pathways address different use cases and should not be conflated.
  • Figure 3: Three interconnected pillars of language-based objectives. Semantic reward specification (§\ref{['sec:pillar1']}) preserves human intent through natural language. Dynamic adaptation (§\ref{['sec:pillar2']}) enables continuous refinement based on observed behaviors. Human alignment (§\ref{['sec:pillar3']}) ensures objectives remain interpretable and verifiable. These pillars are mutually reinforcing: semantic specifications enable meaningful adaptation, which in turn maintains alignment through transparency.