Reinforcement Learning without Human Feedback for Last Mile Fine-Tuning of Large Language Models

TL;DR

This paper investigates last-mile fine-tuning of large language models without human preference rankings by applying a PPO-based reinforcement learning framework guided by a reward model trained on positive outputs and diverse negative failure modes. By reusing RLHF tooling in a data-only regime and employing a hybrid model-based/model-free RL strategy with a length penalty, the approach is evaluated on abstractive summarization tasks. Results indicate that RL-based fine-tuning yields cleaner, more semantically aligned summaries and improves evaluation metrics such as BLEURT and la-ROUGE relative to maximum-likelihood fine-tuning, though simple post-processing of ML outputs can bridge much of the gap in some datasets. The framework is general, architecture- and dataset-agnostic, and can be extended to penalize additional undesirable outputs like hallucinations, making it a practical option when human rankings are unavailable.

Abstract

Reinforcement learning is used to align language models with human preference signals after first pre-training the model to predict the next token of text within a large corpus using likelihood maximization. Before being deployed in a specific domain, models are often further fine-tuned on task specific data. Since human preferences are often unavailable for the last step, it is performed using likelihood maximization as that is the typical default method. However, reinforcement learning has other advantages besides facilitating alignment to a human derived reward function. For one, whereas likelihood maximization is a form of imitation learning in which the model is trained on what to do under ideal conditions, reinforcement learning is not limited to demonstrating actions just for optimally reached states and trains a model what to do under a range of scenarios as it explores the policy space. In addition, it also trains a model what not to do, suppressing competitive but poor actions. This work develops a framework for last-mile fine-tuning using reinforcement learning and tests whether it garners performance gains. The experiments center on abstractive summarization, but the framework is general and broadly applicable. Use of the procedure produced significantly better results than likelihood maximization when comparing raw predictions. For the specific data tested, the gap could be bridged by employing post-processing of the maximum likelihood outputs. Nonetheless, the framework offers a new avenue for model optimization in situations where post-processing may be less straightforward or effective, and it can be extended to include more complex classes of undesirable outputs to penalize and train against, such as hallucinations.