Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation

Seamus Somerstep, Vinod Raman, Unique Subedi, Yuekai Sun

TL;DR

The paper provides a formal, information-theoretic comparison of Best-of-N BoN and supervised fine-tuning for bit-string autoregressive tasks, highlighting realizable versus agnostic learning and two inference-time regimes. In the realizable setting, SFT achieves convergence to near-perfect reward with a rate nearly independent of the response length $T$, while BoN incurs a linear-in-$T$ term in its convergence, implying SFT can be preferable when the target function lies within the hypothesis class. In agnostic scenarios, BoN can maintain positive reward under broader conditions, whereas SFT may fail depending on the misalignment between training-time teacher forcing and test-time autoregressive generation. The results quantify precise bounds in terms of VC dimensions and a coverage constant $\alpha$, clarifying when inference-time BoN or training-time SFT will dominate for task adaptation of large language models. These findings illuminate the tradeoffs between reward-based selection and supervised data fitting for efficient task adaptation in LLMs, with implications for design choices in real-world alignment and instruction-following systems.

Abstract

Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, trains a reward model to select good responses from a collection generated by an unaltered base model. If the learning setting is realizable, we find that supervised fine-tuning outperforms BoN through a better dependence on the response length in its rate of convergence. If realizability fails, then depending on the failure mode, BoN can enjoy a better rate of convergence in either n or a rate of convergence with better dependence on the response length.

Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation

TL;DR

The paper provides a formal, information-theoretic comparison of Best-of-N BoN and supervised fine-tuning for bit-string autoregressive tasks, highlighting realizable versus agnostic learning and two inference-time regimes. In the realizable setting, SFT achieves convergence to near-perfect reward with a rate nearly independent of the response length , while BoN incurs a linear-in- term in its convergence, implying SFT can be preferable when the target function lies within the hypothesis class. In agnostic scenarios, BoN can maintain positive reward under broader conditions, whereas SFT may fail depending on the misalignment between training-time teacher forcing and test-time autoregressive generation. The results quantify precise bounds in terms of VC dimensions and a coverage constant , clarifying when inference-time BoN or training-time SFT will dominate for task adaptation of large language models. These findings illuminate the tradeoffs between reward-based selection and supervised data fitting for efficient task adaptation in LLMs, with implications for design choices in real-world alignment and instruction-following systems.

Abstract

Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, trains a reward model to select good responses from a collection generated by an unaltered base model. If the learning setting is realizable, we find that supervised fine-tuning outperforms BoN through a better dependence on the response length in its rate of convergence. If realizability fails, then depending on the failure mode, BoN can enjoy a better rate of convergence in either n or a rate of convergence with better dependence on the response length.

Paper Structure

This paper contains 25 sections, 19 theorems, 69 equations.

Table of Contents

  1. Introduction
  2. Problem Set Up and Results
  3. Generation-Verification
  4. Supervised Fine Tuning
  5. Summary of Findings
  6. Related Work
  7. Analysis of BoN
  8. Analysis of SFT with deterministic responses
  9. SFT Fails if target does not lie in class
  10. Rate of convergence for SFT in the Realizable setting
  11. SFT with random responses
  12. Discussion and Limitations
  13. Preliminaries
  14. Binary Classification
  15. Growth Functions and Sample Complexity
  16. ...and 10 more sections

Key Result

Theorem 2.4

Let $\mathcal{F}$ and $P_x$ be any class and marginal distribution over $\Sigma^{*}$ respectively. If $f_{*} \in \mathcal{F}$, then

Theorems & Definitions (36)

  • Example 2.1
  • Definition 2.2
  • Definition 2.3
  • Theorem 2.4: Informal, Realizable setting
  • Theorem 2.5: Informal, Agnostic setting where $\pi_*(\mathbf{y}|x) = f^{\text{AR}}_*(x)$ but $f_* \not \in \mathcal{F}$
  • Theorem 2.6: Informal, Agnostic setting where $f_* \in \mathcal{F}$ but $\pi_*(\mathbf{y}|x) \neq f^{\text{AR}}_*(x)$
  • Theorem 3.2
  • Corollary 3.3
  • Corollary 3.4
  • Proposition 3.5
  • ...and 26 more