Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability

TL;DR

The paper addresses how post-training can impair in-context steerability, output coverage, and distributional alignment in LLMs. It introduces Spectrum Suite as a large, distribution-focused benchmark and Spectrum Tuning as a meta-learning finetuning method that trains to approximate target output distributions using in-context demonstrations. Across three model families, Spectrum Tuning often matches or surpasses pretrained baselines in steerability, diversity, and distributional alignment, while preserving general capabilities better than typical instruction-tuning. These results suggest Spectrum Tuning can enhance user-driven steering and robust distributional modeling, with broad implications for deploying models that respect varied preferences and distributions while maintaining safe and useful behavior.

Abstract

Language model post-training has enhanced instruction-following and performance on many downstream tasks, but also comes with an often-overlooked cost on tasks with many possible valid answers. We characterize three desiderata for conditional distributional modeling: in-context steerability, valid output space coverage, and distributional alignment, and document across three model families how current post-training can reduce these properties. In particular, we disambiguate between two kinds of in-context learning: ICL for eliciting existing underlying knowledge or capabilities, and in-context steerability, where a model must use in-context information to override its priors and steer to a novel data generating distribution. To better evaluate and improve these desiderata, we introduce Spectrum Suite, a large-scale resource compiled from >40 data sources and spanning >90 tasks requiring models to steer to and match diverse distributions ranging from varied human preferences to numerical distributions and more. We find that while current post-training techniques help elicit underlying capabilities and knowledge, they hurt models' ability to flexibly steer in-context. To mitigate these issues, we propose Spectrum Tuning, a post-training method using Spectrum Suite to improve steerability and distributional coverage. We find that Spectrum Tuning often improves over pretrained models and their instruction-tuned counterparts, enhancing steerability, spanning more of the output space, and improving distributional alignment on held-out datasets.

Figures (15)

• Figure 1: Three desiderata for conditional distributional modeling. Example outputs and data are drawn from google/gemma-3-12b.
• Figure 2: Example tasks from Spectrum Suite in the format used for Spectrum Tuning. In our method, we shuffle the data, put it into the above format, and finetune with cross-entropy loss only on the (highlighted) output tokens, including the terminal token.
• Figure 3: Task composition from Spectrum Suite. Individual modeling tasks (data from the same person) are shaded.
• Figure 4: Change in accuracy on Spectrum Suite from the pretrained to instruction-tuned model. Current instruction-tuning hurts in-context steerability.
• Figure 5: Current instruction-tuning generally helps on capability benchmarks.