A Call for Clarity in Beam Search: How It Works and When It Stops

TL;DR

This paper uncovers a hidden FCFS stopping rule in popular beam decoding implementations and introduces a simple one-line modification, the patience factor $p$, to decouple search breadth from depth. By generalizing the stopping criterion, the authors enable deeper or shallower search at fixed beam size $k$ and demonstrate performance gains on news summarization and machine translation with negligible slowdown. They also show that different beam decoding variants can yield large performance differences, underscoring the need for clear reporting of decoding strategies in research. The work offers a practical, minimal-change solution and calls for standardizing beam search descriptions to improve reproducibility.

Abstract

Text generation with beam search has proven successful in a wide range of applications. We point out that, though largely overlooked in the literature, the commonly-used implementation of beam decoding (e.g., Hugging Face Transformers and fairseq) uses a first come, first served heuristic: it keeps a set of already completed sequences over time steps and stops when the size of this set reaches the beam size. Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search. Empirical results demonstrate that adjusting this patience factor improves decoding performance of strong pretrained models on news text summarization and machine translation over diverse language pairs, with a negligible inference slowdown. Our approach only modifies one line of code and can be thus readily incorporated in any implementation. Further, we find that different versions of beam decoding result in large performance differences in summarization, demonstrating the need for clarity in specifying the beam search implementation in research work. Our code will be available upon publication.

Figures (6)

• Figure 1: FCFS Beam Decoding with Controlled Patience
• Figure 2: Vanilla Beam Decoding
• Figure 3: FCFS with patience factor $p$ vs. vanilla beam decoding. $k$ denotes the beam size. FCFS stores finished sentences in $F$, but they stay in (and later may fall off from) beam $B$ during vanilla decoding. $k\! \cdot\! p$ determines the size of $F$. The illustration of beam decoding here is inspired by huang-etal-2012-structured.
• Figure 4: Effects of varying patience factors $p$ on the dev. score (A and B) and inference speed (C). The inference speed is measured with batch size 20, relative to the vanilla version on the same single Nvidia A100-SXM GPU. Other languages pairs were similar to EN$\rightarrow$DE (A). CNNDM also had similar trends to XSUM (B).
• Figure 5: Effects of controlled patience on the dev. data over varying beam sizes. The length penalty value is 1. We evaluate with COMET for machine translation and ROUGE-L for XSUM summarization.