Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarm

Hyeonjun Kim, Kanghoon Lee, Junho Park, Jiachen Li, Jinkyoo Park

TL;DR

This work tackles the challenge of aligning multi-agent USV swarm policies with human preferences by extending reinforcement learning from a human-in-the-loop perspective. It introduces an Agent-Level Feedback framework that uses a Bradley-Terry reward model built on a GNN-GRU backbone, and couples this with IPPO-based fine-tuning to produce policies that satisfy user preferences without sacrificing task performance. An LLM-based evaluator validates feedback scenarios such as region constraints, collision avoidance, and task allocation, while extensive experiments in a pursuit–evasion MARL setting demonstrate improved preference satisfaction and robust reward learning. The approach advances practical adaptability of USV swarms by enabling granular credit assignment and robust policy refinement, with future work aiming to reduce reliance on large-scale human feedback through active learning.

Abstract

Multi-Agent Reinforcement Learning (MARL) has shown promise in solving complex problems involving cooperation and competition among agents, such as an Unmanned Surface Vehicle (USV) swarm used in search and rescue, surveillance, and vessel protection. However, aligning system behavior with user preferences is challenging due to the difficulty of encoding expert intuition into reward functions. To address the issue, we propose a Reinforcement Learning with Human Feedback (RLHF) approach for MARL that resolves credit-assignment challenges through an Agent-Level Feedback system categorizing feedback into intra-agent, inter-agent, and intra-team types. To overcome the challenges of direct human feedback, we employ a Large Language Model (LLM) evaluator to validate our approach using feedback scenarios such as region constraints, collision avoidance, and task allocation. Our method effectively refines USV swarm policies, addressing key challenges in multi-agent systems while maintaining fairness and performance consistency.

Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarm

TL;DR

This work tackles the challenge of aligning multi-agent USV swarm policies with human preferences by extending reinforcement learning from a human-in-the-loop perspective. It introduces an Agent-Level Feedback framework that uses a Bradley-Terry reward model built on a GNN-GRU backbone, and couples this with IPPO-based fine-tuning to produce policies that satisfy user preferences without sacrificing task performance. An LLM-based evaluator validates feedback scenarios such as region constraints, collision avoidance, and task allocation, while extensive experiments in a pursuit–evasion MARL setting demonstrate improved preference satisfaction and robust reward learning. The approach advances practical adaptability of USV swarms by enabling granular credit assignment and robust policy refinement, with future work aiming to reduce reliance on large-scale human feedback through active learning.

Abstract

Multi-Agent Reinforcement Learning (MARL) has shown promise in solving complex problems involving cooperation and competition among agents, such as an Unmanned Surface Vehicle (USV) swarm used in search and rescue, surveillance, and vessel protection. However, aligning system behavior with user preferences is challenging due to the difficulty of encoding expert intuition into reward functions. To address the issue, we propose a Reinforcement Learning with Human Feedback (RLHF) approach for MARL that resolves credit-assignment challenges through an Agent-Level Feedback system categorizing feedback into intra-agent, inter-agent, and intra-team types. To overcome the challenges of direct human feedback, we employ a Large Language Model (LLM) evaluator to validate our approach using feedback scenarios such as region constraints, collision avoidance, and task allocation. Our method effectively refines USV swarm policies, addressing key challenges in multi-agent systems while maintaining fairness and performance consistency.

Paper Structure

This paper contains 15 sections, 6 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. USV Swarm Control
  4. Preference-based Reinforcement Learning
  5. Problem Formulation
  6. Methods
  7. Agent-Level Feedback within an Episode
  8. Agent-wise Reward Learning
  9. Fine-Tuning the MARL Policy
  10. Experiments
  11. Experimental Setup
  12. Reward Model Evaluation
  13. MARL Policy Fine-tuning
  14. Ablation Study
  15. Conclusion

Figures (5)

  • Figure 1: Illustration of the MARL policy fine-tuning process with human feedback. The base policy achieves task success but fails to satisfy preferences due to suboptimal agent performance (e.g., Agent 2). The fine-tuned policy improves both task success and preference satisfaction, demonstrating enhanced agent behavior guided by human preference.
  • Figure 2: A diagram of the proposed method for fine-tuning MARL policies through Agent-Level human feedback.
  • Figure 3: Comparison of reward distribution between team-level and agent-level feedback for InCircle and Crossing scenarios.
  • Figure 4: Comparison of USV swarm behaviors before and after fine-tuning. (a) Base policy before fine-tuning. (b)–(d) Fine-tuned policies incorporating human feedback for each criterion: InCircle, Crossing, and Assignment.
  • Figure 5: Trade-off between task performance and human preference across different $\lambda$ for two criteria. The grey-shaded region indicates the base model.