Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Object Detection for Autonomous Driving via Curriculum-Guided Group Relative Policy Optimization

Xu Jia

TL;DR

This work tackles robust multimodal object detection for autonomous driving by marrying Group Relative Policy Optimization (GRPO) with curriculum-guided data scheduling and difficulty-aware filtering to address sparse and noisy reward signals. The approach uses a multi-component loss (IoU, Format, KL) and intra-group reward normalization to stabilize learning while progressively presenting harder samples through a curriculum. Empirical results on CODA and BDD-100K show substantial gains in IoU for critical categories (e.g., 9.4 percentage points on BDD-100K pedestrians/non-motorized vehicles and 7.1 percentage points on CODA traffic participants), demonstrating improved localization and robustness across domains. The findings suggest reinforcement-driven optimization with structured data curricula as a scalable path toward robust, interpretable vision-language perception in real-world autonomous systems.

Abstract

Multimodal Large Language Models (MLLMs) excel in vision-language reasoning but often struggle with structured perception tasks requiring precise localization and robustness. We propose a reinforcement learning framework that augments Group Relative Policy Optimization (GRPO) with curriculum-based data scheduling and difficulty-aware filtering. This approach stabilizes optimization under sparse, noisy rewards and enables progressive adaptation to complex samples. Evaluations on autonomous driving benchmarks demonstrate substantial improvements in detection accuracy and robustness. Ablation studies confirm the importance of reward design, KL regularization, and curriculum pacing for convergence stability and generalization. Our findings highlight reinforcement-driven optimization with structured data curricula as a scalable path toward robust and interpretable multimodal detection.

Robust Object Detection for Autonomous Driving via Curriculum-Guided Group Relative Policy Optimization

TL;DR

This work tackles robust multimodal object detection for autonomous driving by marrying Group Relative Policy Optimization (GRPO) with curriculum-guided data scheduling and difficulty-aware filtering to address sparse and noisy reward signals. The approach uses a multi-component loss (IoU, Format, KL) and intra-group reward normalization to stabilize learning while progressively presenting harder samples through a curriculum. Empirical results on CODA and BDD-100K show substantial gains in IoU for critical categories (e.g., 9.4 percentage points on BDD-100K pedestrians/non-motorized vehicles and 7.1 percentage points on CODA traffic participants), demonstrating improved localization and robustness across domains. The findings suggest reinforcement-driven optimization with structured data curricula as a scalable path toward robust, interpretable vision-language perception in real-world autonomous systems.

Abstract

Multimodal Large Language Models (MLLMs) excel in vision-language reasoning but often struggle with structured perception tasks requiring precise localization and robustness. We propose a reinforcement learning framework that augments Group Relative Policy Optimization (GRPO) with curriculum-based data scheduling and difficulty-aware filtering. This approach stabilizes optimization under sparse, noisy rewards and enables progressive adaptation to complex samples. Evaluations on autonomous driving benchmarks demonstrate substantial improvements in detection accuracy and robustness. Ablation studies confirm the importance of reward design, KL regularization, and curriculum pacing for convergence stability and generalization. Our findings highlight reinforcement-driven optimization with structured data curricula as a scalable path toward robust and interpretable multimodal detection.

Paper Structure

This paper contains 6 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Group Relative Policy Optimization
  5. Object Detection for Autonomous Driving
  6. Method

Figures (4)

  • Figure :
  • Figure :
  • Figure : Figure 2-1: Our training pipeline begins with RefCOCO image samples, where the InternVL model is optimized using GRPO with Accuracy-IoU as the reward, establishing fundamental grounding capabilities. Subsequently, CODA and BDD-100K datasets are introduced. For each sample, the model generates eight candidate outputs, which are filtered via mean and variance of IoU to assess difficulty levels. A curriculum learning scheme is then applied, progressively advancing from easier to harder subsets. Finally, the trained model is evaluated on real-world test sets.
  • Figure : Figure2-2 Overview of GRPO: candidate responses are sampled from the policy distribution, rewards are standardized within each group, and relative advantages guide policy updates under a reference constraint.