Reverse Engineering Human Preferences with Reinforcement Learning

TL;DR

This work reveals a vulnerability in the LLM-as-a-judge paradigm: upstream preambles, optimized via reinforcement learning, can steer frozen candidate LLMs to receive higher judge scores without altering the final text post hoc. By training a dedicated preamble generator with Contrastive Policy Gradient against a judge-LLM reward, the approach (RLRE) achieves transferability across candidate-LLMs, judge-LLMs, and benchmarks, while remaining difficult to detect with standard perplexity checks or human inspection. The findings question the reliability of current LLM-evaluation schemes and demonstrate a flexible, plug-and-play method to optimize inputs upstream of model generation, with potential applications beyond adversarial attacks, such as toxicity or bias mitigation. The work underscores the need for robust, leakage-resistant evaluation frameworks and suggests that evaluating human preferences via LLMs will require careful safeguards and diverse, ensemble-based judgments.

Abstract

The capabilities of Large Language Models (LLMs) are routinely evaluated by other LLMs trained to predict human preferences. This framework--known as LLM-as-a-judge--is highly scalable and relatively low cost. However, it is also vulnerable to malicious exploitation, as LLM responses can be tuned to overfit the preferences of the judge. Previous work shows that the answers generated by a candidate-LLM can be edited post hoc to maximise the score assigned to them by a judge-LLM. In this study, we adopt a different approach and use the signal provided by judge-LLMs as a reward to adversarially tune models that generate text preambles designed to boost downstream performance. We find that frozen LLMs pipelined with these models attain higher LLM-evaluation scores than existing frameworks. Crucially, unlike other frameworks which intervene directly on the model's response, our method is virtually undetectable. We also demonstrate that the effectiveness of the tuned preamble generator transfers when the candidate-LLM and the judge-LLM are replaced with models that are not used during training. These findings raise important questions about the design of more reliable LLM-as-a-judge evaluation settings. They also demonstrate that human preferences can be reverse engineered effectively, by pipelining LLMs to optimise upstream preambles via reinforcement learning--an approach that could find future applications in diverse tasks and domains beyond adversarial attacks.

Figures (4)

• Figure 1: Reinforcement Learning for Reverse Engineering (RLRE) pipeline for training a preamble generator. Given a question $q$ from a training set, we prepend to it a general instruction and feed it to the preamble generator $\pi$. In order for $\pi$ to learn the policy, we sample two preambles per question, $p$ and $p'$. The respective rewards are obtained by appending $q$ to $p$ and $p'$, respectively, and (i) passing each as input to the candidate-LLM, which generates the responses $c$ and $c'$, and (ii) having the judge-LLM evaluate each question-response pair and extracting the respective numerical rewards from these evaluations. The loss function that optimises $\pi$ depends on the delta between the rewards $R(q, c)$ and $R(q, c')$.
• Figure 2: Average scores per question type obtained by candidate-LLMs using a refinement-aware bias attack, raina-etal-2024-llm's universal adversarial attack, and the adversarial preamble generator.
• Figure 3: Proportion of LLM responses that have been labelled as 'attacked' by human evaluators. Responses generated using adversarial preambles are identified as attacked nearly as rarely as those produced by a non-attacked model.
• Figure 4: False negative rate (FNR) of PPL-W for each attack type. Verbosity bias, bandwagon bias and our preamble-based attack are rarely detected with this method (FNR $\geq 0.90$). The universal adversarial attack raina-etal-2024-llm is almost always detected (FNR $=0.04$).