Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization

TL;DR

This work tackles the high labor cost of RLHF by introducing STR, a self-supervised text ranking pipeline that enables PPO-based fine-tuning of pre-trained language models without human annotations. It combines TextRank-based semantic ranking with ISODATA clustering to produce representative contrasting data and trains a reward model to steer generation toward higher-quality outputs. Across GPT-2 and GPT-Neo on DailyDialogue, Cornell Movie, and SQuAD v2.0, STR improves BLEU, GLEU, METEOR, and EM metrics while manual evaluation shows substantial agreement with human judgments, indicating reliable self-correction signals. The approach substantially reduces training costs and demonstrates the feasibility of self-supervised, PPO-guided fine-tuning for scalable, reliable language models.

Abstract

Wide usage of ChatGPT has highlighted the potential of reinforcement learning from human feedback. However, its training pipeline relies on manual ranking, a resource-intensive process. To reduce labor costs, we propose a self-supervised text ranking approach for applying Proximal-Policy-Optimization to fine-tune language models while eliminating the need for human annotators. Our method begins with probabilistic sampling to encourage a language model to generate diverse responses for each input. We then employ TextRank and ISODATA algorithms to rank and cluster these responses based on their semantics. Subsequently, we construct a reward model to learn the rank and optimize our generative policy. Our experimental results, conducted using two language models on three tasks, demonstrate that the models trained by our method considerably outperform baselines regarding BLEU, GLEU, and METEOR scores. Furthermore, our manual evaluation shows that our ranking results exhibit a remarkably high consistency with that of humans. This research significantly reduces training costs of proximal policy-guided models and demonstrates the potential for self-correction of language models.

Figures (2)

• Figure 1: Our pipeline comprises three steps: 1) fine-tuning a language model to generate multiple candidate answers for a given question and using the TextRank algorithm to rank these answers; 2) filtering out non-representative answers using the ISODATA algorithm and training a reward model based on the remaining answers, and 3) scoring the ranked answers using the reward model and updating the generation policy via PPO. Note that we implemented the generative policy through a PLM in our study.
• Figure 2: We applied principal component analysis and singular value decomposition to the BERT embeddings of answers generated by GPT-2.