BoN Appetit Team at LeWiDi-2025: Best-of-N Test-time Scaling Can Not Stomach Annotation Disagreements (Yet)

TL;DR

This work investigates whether test-time scaling can mitigate annotation disagreements in NLP by transferring methods from domains with verifiable answers. It evaluates three strategies—Model Averaging, Majority Voting, and Best-of-N sampling with step-wise scores—on LeWiDi-2025 datasets (CSC, MP, PAR, VEN). The study introduces a prediction diversity metric and shows that Model Averaging and Majority Voting consistently improve soft-label and perspectivist predictions, while Best-of-N sampling fails to transfer from math to LeWiDi tasks. The findings imply that current reasoning-focused post-training may need adaptation to handle interpretative variability in NLP, guiding future work toward training on similar tasks or refining step-wise evaluation to better capture diverse perspectives.

Abstract

Test-time scaling is a family of techniques to improve LLM outputs at inference time by performing extra computation. To the best of our knowledge, test-time scaling has been limited to domains with verifiably correct answers, like mathematics and coding. We transfer test-time scaling to the LeWiDi-2025 tasks to evaluate annotation disagreements. We experiment with three test-time scaling methods: two benchmark algorithms (Model Averaging and Majority Voting), and a Best-of-N sampling method. The two benchmark methods improve LLM performance consistently on the LeWiDi tasks, but the Best-of-N method does not. Our experiments suggest that the Best-of-N method does not currently transfer from mathematics to LeWiDi tasks, and we analyze potential reasons for this gap.

Figures (9)

• Figure 1: Best-of-N sampling with step-wise scores. For each problem a reasoning LLM generates $N$ soft-labels and CoT. Next, an LLM-as-a-judge scores each step in the CoT for correctness, and BoN selects the soft-label with the highest total score. Takeaway: Sampling multiple times increases the chances for a good prediction.
• Figure 2: Test-time Scaling Benchmarks. Top: soft-label task. Bottom: perspectivist task. Distance metric on the x-axis (lower is better). Model Averaging and Majority Voting (green) are consistently better than Simple Sampling (orange) in the soft-label and perspectivist tasks, respectively.
• Figure 3: Best-of-N Sampling on LeWiDi Tasks. Top: soft-label task. Bottom: perspectivist task. Distance metric on the y-axis (lower is better). Higher $N$ should lead to better performance, but does not. No consistent pattern emerges across datasets and tasks. In red are the test-time scaling benchmarks, which BoN generally does not beat. The shaded areas show the 0.25 and 0.75 quantiles.
• Figure 4: Model performance (lower is better) varies with prediction diversity and is related to the difficulty of the problem.
• Figure 5: High prediction diversity leads to better performance of test-time scaling methods. Both the upper bound (BoN oracle) and Model Averaging benefit from higher prediction diversity. The y-axis shows the improvement over Simple Sampling.