Reducing the Scope of Language Models

TL;DR

The paper tackles the challenge of scoping large language models to domain-specific tasks by forcing generation only for relevant queries and rejecting irrelevant ones. It evaluates a range of techniques—system prompting, supervised fine-tuning, direct preference optimization, probing-based classifiers, and Circuit Breakers—across three model families and multiple task categories, using Accept Score and rejection metrics on in- and out-of-distribution data. Key findings show that supervised fine-tuning shines when rejection data are diverse, while Circuit Breakers perform best under low-diversity conditions or adversarial prompts; layering SFT with CB often yields robust, practical performance. The work provides practical guidance for practitioners on selecting, combining, and tuning scoping methods for real deployments, including insights on data diversity, model scale, and interpretation of internal representations.

Abstract

Large language models (LLMs) are deployed in a wide variety of user-facing applications. Typically, these deployments have some specific purpose, like answering questions grounded on documentation or acting as coding assistants, but they require general language understanding. In such deployments, LLMs should respond only to queries that align with the intended purpose and reject all other requests, such as generating poetry or answering questions about physics, a task we refer to as `scoping'. We conduct a comprehensive empirical evaluation of various methods, ranging from prompting, fine-tuning to preference learning and the recently proposed general alignment technique known as Circuit Breakers (CB). Across three families of language models and a broad variety of tasks, we show that it is possible to scope language models. We examine scoping for multiple topics, and fine-grained topics. We ablate diversity of irrelevant queries, layer different techniques, conduct adversarial evaluations and more. Among other results, we find that when diverse examples of irrelevant queries are available, simple supervised fine-tuning produces the best results, but when such diversity is low, Circuit Breakers perform quite well. One can often get the benefits of both methods by layering them in succession. We intend our study to serve as a practitioner's guide to scoping LLMs.

Figures (14)

• Figure 1: We study the ability to scope language models to specific topics. We assume access to a set of relevant (accept) queries and irrelevant (reject) queries, where the accept queries correspond to a relatively narrow domain. We examine how well different methods cause the language model to accept only the relevant examples, while rejecting all other examples, including out-of-distribution requests that weren't seen during training.
• Figure 2: Scoping across different language models. We see that system prompting is insufficient, and different methods have different success rates for different models. Clearly it is possible to scope language models to particular distributions.
• Figure 3: Results for increasing diversity of rejection set. We see across models that CB performs relatively better than SFT when data diversity is low, but SFT is much stronger with more rejections sets. Probing appears strong across the board, though sometimes leads to overrejection on the Accept set.
• Figure 4: Cosine distance between base model representations and LoRA tuned model. Rows represent model layer, columns represent tail position of prompt, averaged over all queries in the dataset. Top row: Accept data. Middle row: In distribution reject data. Bottom row: Out of distribution reject data. We see that CB-based methods make changes to representations across the context, while DPO and SFT only change the tail of the context. Layering SFT and CB yields both effects.
• Figure 5: Results for increasing diversity of rejection set on tasks besides sentiment analysis (SA). Like in the SA case we see that CB performs worse with more diverse data.