Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models

TL;DR

PGSRM proposes embedding-based semantic rewards as an alternative to RLHF for teacher-guided alignment of small transformer models. By using a fixed parent to generate reference outputs and a dense cosine-similarity reward in embedding space, the method eliminates the need for labeled preferences or a separate reward model. Across five tasks and two GPT-2 sizes, PGSRM delivers smoother reward trajectories and more stable PPO dynamics than a binary baseline, suggesting improved sample efficiency and optimization stability. The approach demonstrates the potential of semantic shaping as a lightweight alignment primitive, while acknowledging inherent biases in the teacher and embedding space that require careful evaluation.

Abstract

We introduce the Parent-Guided Semantic Reward Model (PGSRM), a lightweight reward framework for reinforcement learning (RL) of transformer language models. PGSRM replaces binary correctness signals, human preference data, and trained reward models with a simple signal: cosine similarity between a parent model's reference output embedding and a child model's generated output for the same input. This yields a dense, semantically meaningful reward with no human annotation or additional model training. We apply PGSRM on five language tasks and find that it produces smoother reward improvement and more stable PPO dynamics than a binary reward baseline, suggesting that embedding-based semantic rewards are a practical alternative to RLHF-style reward modeling for parent-guided alignment in smaller transformer models.

Figures (5)

• Figure 1: Training dynamics for the color mixing task.
• Figure 2: Training dynamics for the antonym generation task.
• Figure 3: Training dynamics for the word categorization task.
• Figure 4: Training dynamics for the exact-string copying task.
• Figure 5: Training dynamics for the sentiment inversion task.