A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

TL;DR

This paper addresses the high cost and limited scalability of human-in-the-loop RLHF by introducing CAGSR, a self-supervised reinforcement learning framework that leverages cross-attention signals within Transformer-based LLMs to guide fine-tuning without human feedback. It defines a reward composed of prompt coverage, attention focus, and repetition penalties, and optimizes the policy via PPO to maximize the expected self-supervised reward. Empirical results on synthetic QA and instruction datasets show that CAGSR improves prompt relevance and coherence relative to no-RL and synthetic-preference baselines, though it does not yet reach fully human-supervised RLHF performance. The work demonstrates a scalable, cost-effective direction for alignment and suggests hybrid approaches that combine cross-attention rewards with limited human input to further close the gap to human-labeling baselines. Overall, CAGSR offers a practical path toward scalable alignment of large language models with reduced human annotation requirements while maintaining competitive output quality.

Abstract

We propose a novel reinforcement learning framework for post training large language models that does not rely on human in the loop feedback. Instead, our approach uses cross attention signals within the model itself to derive a self supervised reward, thereby guiding iterative fine tuning of the model policy. By analyzing how the model attends to the input prompt during generation, we construct measures of prompt coverage, focus, and coherence. We then use these measures to rank or score candidate responses, providing a reward signal that encourages the model to produce well aligned, on topic text. In empirical comparisons against standard policy gradient methods and RL fine tuning with synthetic preference models, our method shows significant gains in prompt relevance and consistency over a non RL baseline. While it does not yet match the performance of fully human supervised RLHF systems, it highlights an important direction for scaling alignment with minimal human labeling. We provide a detailed analysis, discuss potential limitations, and outline future work for combining cross-attention based signals with smaller amounts of human feedback.

Figures (1)

• Figure 1: Figure X illustrates the complete pipeline of the proposed Cross-Attention–Guided Self-Reinforcement (CAGSR) framework for fine-tuning large language models without human-in-the-loop feedback. The process commences with an input prompt x (which can be an instruction or question) that is fed into the current model policy $\pi_\theta(y \mid x)$ to generate multiple candidate responses using sampling techniques such as top-k or nucleus sampling. As the candidate responses are produced, the model’s decoder extracts cross-attention maps from its final layers, capturing how each generated token distributes its attention across the input prompt. These attention maps are used to compute a composite reward R(x,y) that reflects three key qualities: the $\emph{coverage}$ of important prompt tokens, the $\emph{focus}$ (or sharpness) of the attention distributions (evaluated via their negative entropy), and a $\emph{repetition penalty}$ that discourages degenerate, repetitive outputs. An advantage estimate $A(x,y) = R(x,y) - V_{\theta_{\text{old}}}(x)$ is then calculated using a baseline value function, and this estimate is incorporated into a Proximal Policy Optimization (PPO) update that refines the model’s policy. The updated policy is iteratively looped back into the candidate generation process, progressively steering the model toward producing more coherent, on-topic, and well-aligned responses. This figure captures the self-supervised feedback loop that underpins CAGSR, emphasizing its potential to reduce dependency on costly human annotation while maintaining a high degree of output quality.