Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position: Capability Control Should be a Separate Goal From Alignment

Shoaib Ahmed Siddiqui, Eleni Triantafillou, David Krueger, Adrian Weller

TL;DR

This paper argues that capability control should be treated as a distinct objective from alignment rather than a mere byproduct of post-training alignment. It proposes a defense-in-depth framework spanning data-based, learning-based, and system-based controls to harden foundation models against misuse and failure, detailing concrete mechanisms such as data filtering, RLHF-based refinement, adversarial training, system prompts, and guardrails. It discusses open challenges including evaluation, open-weight models, dual-use knowledge, and compositional generalization, and advocates coordinated, multi-layer research to reduce risk while preserving utility. The work highlights the practical importance of combining complementary approaches across the model lifecycle to provide stronger, more deterministic guarantees in high-stakes deployments.

Abstract

Foundation models are trained on broad data distributions, yielding generalist capabilities that enable many downstream applications but also expand the space of potential misuse and failures. This position paper argues that capability control -- imposing restrictions on permissible model behavior -- should be treated as a distinct goal from alignment. While alignment is often context and preference-driven, capability control aims to impose hard operational limits on permissible behaviors, including under adversarial elicitation. We organize capability control mechanisms across the model lifecycle into three layers: (i) data-based control of the training distribution, (ii) learning-based control via weight- or representation-level interventions, and (iii) system-based control via post-deployment guardrails over inputs, outputs, and actions. Because each layer has characteristic failure modes when used in isolation, we advocate for a defense-in-depth approach that composes complementary controls across the full stack. We further outline key open challenges in achieving such control, including the dual-use nature of knowledge and compositional generalization.

Position: Capability Control Should be a Separate Goal From Alignment

TL;DR

This paper argues that capability control should be treated as a distinct objective from alignment rather than a mere byproduct of post-training alignment. It proposes a defense-in-depth framework spanning data-based, learning-based, and system-based controls to harden foundation models against misuse and failure, detailing concrete mechanisms such as data filtering, RLHF-based refinement, adversarial training, system prompts, and guardrails. It discusses open challenges including evaluation, open-weight models, dual-use knowledge, and compositional generalization, and advocates coordinated, multi-layer research to reduce risk while preserving utility. The work highlights the practical importance of combining complementary approaches across the model lifecycle to provide stronger, more deterministic guarantees in high-stakes deployments.

Abstract

Foundation models are trained on broad data distributions, yielding generalist capabilities that enable many downstream applications but also expand the space of potential misuse and failures. This position paper argues that capability control -- imposing restrictions on permissible model behavior -- should be treated as a distinct goal from alignment. While alignment is often context and preference-driven, capability control aims to impose hard operational limits on permissible behaviors, including under adversarial elicitation. We organize capability control mechanisms across the model lifecycle into three layers: (i) data-based control of the training distribution, (ii) learning-based control via weight- or representation-level interventions, and (iii) system-based control via post-deployment guardrails over inputs, outputs, and actions. Because each layer has characteristic failure modes when used in isolation, we advocate for a defense-in-depth approach that composes complementary controls across the full stack. We further outline key open challenges in achieving such control, including the dual-use nature of knowledge and compositional generalization.
Paper Structure (33 sections, 2 figures)

This paper contains 33 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Capability Control
  3. Objective of Capability Control
  4. Realization of Capability Control
  5. Does Alignment Cover Capability Control?
  6. Difference between Capability Suppression and Removal
  7. Data-Based Capability Control
  8. Data Filtering
  9. Data Curation
  10. Synthetic Data Generation
  11. Learning-Based Capability Control
  12. Categorization based on Training Objective
  13. Learning from Behavior Demonstrations
  14. Learning from Preference Data
  15. Unlearning Undesired Behaviors
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Overview of Capability Control Layers. Capability control can be applied at three complementary levels across the model lifecycle: (i) data-based control shapes the training distribution of foundation models; (ii) learning-based control modifies model weights or representations during training; and (iii) system-based control constrains the behavior of the deployed system via guardrails on inputs, outputs, and tool access.
  • Figure 2: System-Based Capability Control. This layer enforces control at inference time via wrappers on the system's inputs and outputs. These mechanisms can intercept, block, redact, or route traffic based on safety policies and monitoring signals, operating independently of the model's internal weights and representations.