A-VERT: Agnostic Verification with Embedding Ranking Targets

TL;DR

A-VERT addresses the challenge of evaluating free-form LM QA responses under real-world conditions by proposing an agnostic verification framework that ranks candidate target groups using semantic embeddings or rerankers. By constructing correct and wrong semantic groups and applying a normalized ranking mechanism, A-VERT achieves high agreement with human annotations across multiple benchmarks while using small parameter-efficient models. The method demonstrates strong correlations with human judgments (R^2 ≈ 0.967) and outperforms traditional exact-match and logprob-based scoring, highlighting its practicality for open-ended and multiple-choice QA. The approach enables benchmarks to reflect day-to-day LM usage, reduces reliance on expensive adjudication, and offers avenues for extending to additional targets and domains in future work.

Abstract

The automatic evaluation of Language Model (LM) responses is a critical piece in the development of benchmarks and metrics, both for model training and quality assessment of production model endpoints. The current approaches to response classification relies on methods that are too expensive (i.e. LLM-as-a-Judge) or that are far from real-world conditions (string-matching, logprob). In this paper, a structure-free evaluation method is presented. The method makes use of semantic embedding distances to match target candidates with arbitrary LM-generated text, resulting in a robust classification of the response at a relatively low compute cost (embedding models of less than $10B$ parameters). The results show a regression score of ~0.97 and an accuracy of ~96% against human annotators, tested over 3 data sets and 3 different LM architectures.

Figures (7)

• Figure 1: Overview of the A-VERT methodology. The process begins with a data source containing question, answer, and choices. During prompting, the question and choices are formatted and presented to the LM. The LM generates a free-form response $g$ during inference. In the evaluation phase, A-VERT constructs semantic target groups: a correct (green) and wrong (red) groups containing enhanced versions of the target answer and incorrect choices, respectively. Each candidate $c_i^{(n)}$ in both groups is scored against the generated response $g$ using a ranking function $r(\cdot)$. Finally, the method selects the representative score for each group, and assigns the LM response to the group with higher ranking score.
• Figure 2: Balanced accuracy of the A-VERT method for each model in different settings. The balanced accuracy is calculated over the three benchmarks and three datasets (bAbI, MMLU and MMLU-Pro).
• Figure 3: Box-plot of distances between the A-VERT scores for the correct and wrong target groups for each benchmark response type. The distances are grouped in the confusion matrix labels, lower average value in the distance between the target groups signals less confidence in the ranking scores. A value of zero means equal weight to each group (random choice), a score of $1.0$ means total confidence for the selected target group.
• Figure 4: Comparison of the mean scores in the bAbI task for each of the analyzed methods: exact match, A-VERT and the human tag. \ref{['fig:babi_comparison']}) For different LMs: GPT-OSS 20B, Meta Llama 3.3 70B Instruct, and Qwen3 30B A3B are in blue, orange, and green, respectively. \ref{['fig:babi_comp_task']}) For bAbI subtask scores: exact match, A-VERT, and human tag are represented as red circle, green cross and blue square, respectively. The GPT-OSS 20B results are excluded from this graph to better assess the agreement of the methods.
• Figure 5: Comparison of the mean scores in the MMLU task for each of the analyzed methods: exact match, A-VERT and the human tag. \ref{['fig:mmlu_dist_comparison']}) For different LMs: GPT-OSS 20B, Meta Llama 3.3 70B Instruct, and Qwen3 30B A3B are in blue, orange, and green, respectively. \ref{['fig:mmlu_comparison_task']}) For MMLU subtask scores: exact match, A-VERT, and human tag are represented as red circle, green cross and blue square, respectively. The Qwen3 30B A3B results are excluded from this graph to better asses the agreement of the methods.