What Is Missing: Interpretable Ratings for Large Language Model Outputs

TL;DR

The What Is Missing (WIM) rating system is introduced to produce rankings from natural-language feedback and yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data.

Abstract

Current Large Language Model (LLM) preference learning methods such as Proximal Policy Optimization and Direct Preference Optimization learn from direct rankings or numerical ratings of model outputs, these rankings are subjective, and a single numerical rating chosen directly by a judge is a poor proxy for the quality of natural language, we introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback, WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method without changing the learning algorithm, to compute a WIM rating, a human or LLM judge writes feedback describing what the model output is missing, we embed the output and the feedback with a sentence embedding model and compute the cosine similarity between the resulting vectors, we empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data, we use interpretable in the following limited sense: for each scalar rating, we can inspect the judge's missing-information text that produced it, enabling qualitative debugging of the preference labels.

Figures (8)

• Figure 1: Flowchart of the WIM method. An LLM produces a natural language output $s_1$. $s_1$ is then evaluated by a human or an LLM judge. The judge's goal is to produce $s_2$, a response containing what is missing in $s_1$. Both $s_1$ and $s_2$ are passed through a sentence embedding model to produce high dimensional vectors $S_1$ and $S_2$. The similarity of $S_1$ and $S_2$ is calculated using cosine similarity and the resulting similarity score is the WIM rating. A higher similarity between $S_1$ and $S_2$ implies that there is less missing from the LLM's output. $n$ model outputs are rated by the WIM method and then sorted to produce a ranking. The ranking of the outputs is then passed to a preference learning algorithm.
• Figure 2: 2D visualization of missingness
• Figure 3: Histogram of ratings from the Numerical rating system and the WIM rating system
• Figure 4: Histogram of rating deltas from the Numerical Rating System and the WIM Rating System
• Figure 5: Reward advantage trajectories