Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

Punyajoy Saha, Sudipta Halder, Debjyoti Mondal, Subhadarshi Panda

TL;DR

Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models, is introduced, demonstrating that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.

Abstract

Safety alignment is critical for deploying large language models (LLMs) in real-world applications, yet most existing approaches rely on large human-annotated datasets and static red-teaming benchmarks that are costly, difficult to scale, and slow to adapt to evolving model behaviors. Moreover, overly conservative safety mechanisms can reduce model usefulness by rejecting sensitive but legitimate queries. We introduce Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models. Self-MOA operates as a closed loop that dynamically generates model-specific red team prompts, constructs preference data from model-generated responses, and aligns models via multi-objective preference optimization to jointly optimize for safety and helpfulness. Across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41\% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines. These results demonstrate that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.

Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

TL;DR

Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models, is introduced, demonstrating that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.

Abstract

Safety alignment is critical for deploying large language models (LLMs) in real-world applications, yet most existing approaches rely on large human-annotated datasets and static red-teaming benchmarks that are costly, difficult to scale, and slow to adapt to evolving model behaviors. Moreover, overly conservative safety mechanisms can reduce model usefulness by rejecting sensitive but legitimate queries. We introduce Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models. Self-MOA operates as a closed loop that dynamically generates model-specific red team prompts, constructs preference data from model-generated responses, and aligns models via multi-objective preference optimization to jointly optimize for safety and helpfulness. Across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41\% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines. These results demonstrate that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.
Paper Structure (68 sections, 2 equations, 7 figures, 10 tables, 2 algorithms)

This paper contains 68 sections, 2 equations, 7 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pretrained LLMs and Datasets
  4. Datasets
  5. Models
  6. Safety-Reset Initialization
  7. Method
  8. Red Teaming Dataset Generation
  9. Attack and Evaluation
  10. Red Teaming Data Selection
  11. Dynamic Preference Dataset Generation
  12. Multi-Objective Preference Alignment
  13. Experiments
  14. Hyperparameters
  15. Evaluation Details
  16. ...and 53 more sections

Figures (7)

  • Figure 1: This figure shows the entire pipeline for our weak supervision based alignment method - Self-MOA. The dashed lines indicate the same checkpoint of the Target LLM being utilised. Explanations regarding different components are provided in Section \ref{['sec:method']}.
  • Figure 2: Safety score progression on attack datasets across training stages. Lower scores are better.
  • Figure 3: Helpfulness score progression on attack datasets across training stages. Higher scores are better.
  • Figure 4: Helpfulness score progression on safe datasets across training stages. Higher scores are better.
  • Figure 5: Safety score progression on SaladBench datasets across training stages. Lower scores are better
  • ...and 2 more figures