Start Small, Think Big: Curriculum-based Relative Policy Optimization for Visual Grounding

TL;DR

This work shows that explicit Chain-of-Thought (CoT) reasoning can degrade visual grounding performance when CoT length grows, and that simply increasing data size does not guarantee better results. It introduces Curriculum-based Relative Policy Optimization (CuRPO), which uses CoT length and $gIoU$-based rewards to progressively expose the model to easier-to-harder grounding tasks within a GRPO framework. CuRPO yields consistent improvements across RefCOCO, RefCOCO+, RefCOCOg, and LISA, achieving up to +12.52 mAP on RefCOCO and demonstrating strong performance in few-shot scenarios. The approach provides a principled way to balance reasoning and localization in multimodal learning and shows potential for extending curriculum-GRPO strategies to other vision-language tasks.

Abstract

Chain-of-Thought (CoT) prompting has recently shown significant promise across various NLP and computer vision tasks by explicitly generating intermediate reasoning steps. However, we find that reinforcement learning (RL)-based fine-tuned CoT reasoning can paradoxically degrade performance in Visual Grounding tasks, particularly as CoT outputs become lengthy or complex. Additionally, our analysis reveals that increased dataset size does not always enhance performance due to varying data complexities. Motivated by these findings, we propose Curriculum-based Relative Policy Optimization (CuRPO), a novel training strategy that leverages CoT length and generalized Intersection over Union (gIoU) rewards as complexity indicators to progressively structure training data from simpler to more challenging examples. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and LISA datasets demonstrate the effectiveness of our approach. CuRPO consistently outperforms existing methods, including Visual-RFT, with notable improvements of up to +12.52 mAP on RefCOCO. Moreover, CuRPO exhibits exceptional efficiency and robustness, delivering strong localization performance even in few-shot learning scenarios, particularly benefiting tasks characterized by ambiguous and intricate textual descriptions.The code is released on https://github.com/qyoung-yan/CuRPO.

Figures (6)

• Figure 1: Comparison of visual grounding results with and without CoT. The CoT-guided model produces an incorrect bounding box (red) due to misinterpretation of the textual context. In contrast, the model without CoT successfully identifies the correct furniture for watching TV or a movie (green bounding box).
• Figure 2: (a) Accuracy comparison with and without CoT. (b) Performance comparison between short and long CoTs.
• Figure 3: (a) Correlation coefficients between CoT length and reward. (b) Negative impact of increased CoT length on reward.
• Figure 4: Overview of curriculum-based GRPO training framework for visual grounding. We first sort training examples by the complexity indicated by their CoT length, from simplest (shortest CoT) to hardest (longest CoT). Each query-image pair is fed into a policy model constrained by KL-divergence with a reference model. The model outputs bounding boxes and receives a combined reward incorporating visual accuracy (scaled gIoU) and format correctness. This curriculum strategy progressively guides the model from simpler to increasingly complex visual grounding tasks.
• Figure 5: (Comparison of visual grounding accuracy across different training sample sizes and sorting methods.