BLEUBERI: BLEU is a surprisingly effective reward for instruction following

TL;DR

The paper demonstrates that a simple reference-based metric, $BLEU$, can serve as an effective reward signal for RL-based alignment of LLMs, challenging the need for expensive reward models. By using GRPO to optimize $BLEU$ on a carefully curated 50K-prompt data pool and leveraging high-quality synthetic references, BLEUBERI achieves competitive performance with reward-model-guided RL and SFT across multiple benchmarks and base-model scales. Human evaluation indicates BLEUBERI outputs are on par with reward-model-aligned models and often more factually grounded, while not sacrificing creativity. This work suggests that with quality references, cheap string-matching metrics can be a practical, scalable alternative for instruction-following alignment, opening avenues for reward-design research that reduces reliance on costly human annotation.

Abstract

Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM backbones. Meanwhile, the increasing availability of high-quality synthetic instruction-following datasets raises the question: can simpler, reference-based metrics serve as viable alternatives to reward models during RL-based alignment? In this paper, we show first that BLEU, a basic string-matching metric, surprisingly matches strong reward models in agreement with human preferences on general instruction-following datasets. Based on this insight, we develop BLEUBERI, a method that first identifies challenging instructions and then applies Group Relative Policy Optimization (GRPO) using BLEU directly as the reward function. We demonstrate that BLEUBERI-trained models are competitive with models trained via reward model-guided RL across four challenging instruction-following benchmarks and three different base language models. A human evaluation further supports that the quality of BLEUBERI model outputs is on par with those from reward model-aligned models. Moreover, BLEUBERI models generate outputs that are more factually grounded than competing methods. Overall, we show that given access to high-quality reference outputs (easily obtained via existing instruction-following datasets or synthetic data generation), string matching-based metrics are cheap yet effective proxies for reward models during alignment. We release our code and data at https://github.com/lilakk/BLEUBERI.

Figures (11)

• Figure 1: Human agreement rates for BLEU (with varying numbers of references), two reward models, and other reference-based metrics (with a single Claude reference). BLEU becomes more competitive with reward models as more references are provided, and combining BLEU with a reward model outperforms either alone.
• Figure 2: The highlighted matching $n$-grams in this example show that BLEU can capture correct instruction-following behavior as well as the factuality of the response.
• Figure 3: Factuality results for trained Qwen2.5-7B models across three QA datasets evaluated using VeriScoresong-etal-2024-veriscore. The $K$ values (in parentheses on the x-axis) used for each dataset follow the original paper.
• Figure 4: An example instruction from FreshQA, where red highlights indicate factually incorrect claims. For this instruction, BLEUBERI produces a more factually precise output than GRPO-RM.
• Figure 5: Token counts for difference reference model outputs, prompts, and the two model outputs to be scored.