Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning

Yanchen Xu, Ziheng Jiao, Hongyuan Zhang, Xuelong Li

TL;DR

This paper proposes Group Relative Policy Optimization for Representation Model (GRPO-RM), and investigates the performance of GRPO-like policy in post-training representation models.

Abstract

The Group Relative Policy Optimization (GRPO), a reinforcement learning method used to fine-tune large language models (LLMs), has proved its effectiveness in practical applications such as DeepSeek-R1. It raises a question whether GRPO can be generalized to representation learning models. In this paper, we propose Group Relative Policy Optimization for Representation Model (GRPO-RM), and investigate the performance of GRPO-like policy in post-training representation models. Specifically, our method establishes a predefined output set to functionally replace token sequence sampling in LLMs, thereby generating an output group, which is essential for the probability-driven optimization of GRPO. In addition, a specialized reward function is designed to accommodate the properties of representation models. Extensive experiments are conducted on various real-world datasets to validate the effectiveness of our proposed method.

GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning

TL;DR

This paper proposes Group Relative Policy Optimization for Representation Model (GRPO-RM), and investigates the performance of GRPO-like policy in post-training representation models.

Abstract

The Group Relative Policy Optimization (GRPO), a reinforcement learning method used to fine-tune large language models (LLMs), has proved its effectiveness in practical applications such as DeepSeek-R1. It raises a question whether GRPO can be generalized to representation learning models. In this paper, we propose Group Relative Policy Optimization for Representation Model (GRPO-RM), and investigate the performance of GRPO-like policy in post-training representation models. Specifically, our method establishes a predefined output set to functionally replace token sequence sampling in LLMs, thereby generating an output group, which is essential for the probability-driven optimization of GRPO. In addition, a specialized reward function is designed to accommodate the properties of representation models. Extensive experiments are conducted on various real-world datasets to validate the effectiveness of our proposed method.

Paper Structure

This paper contains 30 sections, 6 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Representation Learning
  4. Downstream Tasks of Visual Representation Learning
  5. Reinforcement Learning Methods for Post-Training
  6. Downstream Tasks Based on DINO
  7. The Proposed Method
  8. Preliminaries
  9. Outputs Sampling for Single Input
  10. Advantages for Representation Learning
  11. Accuracy Rewards
  12. Uniformity Rewards
  13. Additional Punishment for Segmentation
  14. Post-Training Models with GRPO-RM
  15. Experiments
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Framework of GRPO-RM: (a) The post-training architecture of GRPO-RM comprises a base encoder and a task-invariant head. At each epoch, the parameters of the old model ($\theta_{old}$) are updated and incorporated into the loss computation without gradient propagation. Advantages are subsequently computed using ground-truth annotations and the probabilistic distributions generated by the old model. Finally, the loss is derived from the opposite number of Eq. (\ref{['GRPO']}) with hyper-parameter $\beta$ fixed to 0. (b) Specifically, the network of task-specific heads and the tokens used for post-training vary in different tasks. For image classification, we simply feed the class tokens to a full-connected layer-based neural network. For semantic segmentation, the patch tokens are upsampled and projected to obtain a pixel-level prediction.
  • Figure 2: Visualization of DINOv2, Fine-tuning, and GRPO-RM on PASCAL-VOC.
  • Figure 3: Training loss curves for GRPO-RM versus baseline on ImageNet (in-distribution) and Tiny-ImageNet (out-of-distribution). It can be easily derived from the figure that GRPO-RM converge much faster than normal post-training method.