Self-Improvement as Coherence Optimization: A Theoretical Account

TL;DR

This work unifies several unsupervised self-improvement techniques for language models under the notion of coherence optimization, defining coherence as the joint likelihood of context-to-behavior mappings and showing its equivalence to description-length regularization when a pretrained prior is used. It develops a Gibbs-sampling algorithm that converges to a softmax over coherence, and proves that debate, bootstrap, and internal coherence maximization are special cases within this framework. The paper establishes the theoretical optimality of coherence regularization (via KL-optimal priors) and connects it to practical training with pretrained priors, supported by preliminary experiments demonstrating coherence's effectiveness as both an evaluation metric and an optimization objective. Overall, coherence optimization provides a principled mechanism for self-improvement that leverages unlabeled contexts and pretrained knowledge, with implications for truthfulness, robustness, and scalable supervision in large language models.

Abstract

Can language models improve their accuracy without external supervision? Methods such as debate, bootstrap, and internal coherence maximization achieve this surprising feat, even matching golden finetuning performance. Yet why they work remains theoretically unclear. We show that they are all special cases of coherence optimization: finding a context-to-behavior mapping that's most compressible and jointly predictable. We prove that coherence optimization is equivalent to description-length regularization, and that among all such regularization schemes, it is optimal for semi-supervised learning when the regularizer is derived from a pretrained model. Our theory, supported by preliminary experiments, explains why feedback-free self-improvement works and predicts when it should succeed or fail.

Figures (7)

• Figure 1: The coherence optimization framework. The data-generating distribution $\mathcal{D}$ produces supervised samples. Coherence optimization finds $\pi_{\mathrm{SRM}}$ maximizing empirical accuracy with regularization from prior $\mathcal{P}$. A learning system $(\mathcal{M}, \mathcal{A}, \mathcal{S}, \sigma)$ defines the coherence function $\chi$, providing a tractable instance of $\mathcal{P}$ via softmax over coherence: $\mathcal{P}(\pi) = \mathrm X^\beta(\pi) \propto 2^{\beta\chi(\pi)}$. Bayesian inference, in-context learning (ICL), and finetuning are instances of learning systems.
• Figure 2: While greedy decoding gets trapped in local optima, coherence optimization finds the global peak of coherence/compressibility defined by the pretrained prior. Note the distinction between the coherence landscape (determined by the pretrained prior) from the points (deterministic policies in ${\mathcal{A}}^{\mathcal{S}}$).
• Figure 3: Coherence-based metrics detect deceptive solutions while LLM-as-a-Judge is fooled. Blue: honest solutions; Red: deceptive solutions. Left: LLM-as-a-Judge is fooled by deception and assigns higher probabilities (sum of logprobs) to deceptive solutions. Right: The coherence-based metric (PMI) successfully assigns higher scores to honest solutions.
• Figure 4: Mean train accuracy over 40 Gibbs rounds for $\gamma \in \{0.25, 0.45, 0.65, 0.85\}$ (shaded: 95% CI). Higher $\gamma$ holds out a larger context subset $\hat{\mathcal{S}}_t$ when resampling.
• Figure 5: Scaling trends of coherence-truthfulness agreement for different prior policies in an in-context learning system. Lower is better. BASE refers to a pretrained prior, CHAT to a posttrained/RLHF prior, and THINK to a reinforced reasoning prior.

Theorems & Definitions (43)

• Definition 3.1: Deterministic Policy
• Definition 3.2: Learning System
• Example 3.3: Bayesian Learning System
• Example 3.4: In-Context Learning System
• Example 3.5: Finetuning Learning System
• Definition 3.6: Coherence
• Definition 3.7: Softmax Over Coherence
• Example 3.8: Sauces
• Remark 3.9: What is coherence?
• Definition 4.1: Ergodic Learning System