Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations

Swapnaja Achintalwar, Ioana Baldini, Djallel Bouneffouf, Joan Byamugisha, Maria Chang, Pierre Dognin, Eitan Farchi, Ndivhuwo Makondo, Aleksandra Mojsilovic, Manish Nagireddy, Karthikeyan Natesan Ramamurthy, Inkit Padhi, Orna Raz, Jesus Rios, Prasanna Sattigeri, Moninder Singh, Siphiwe Thwala, Rosario A. Uceda-Sosa, Kush R. Varshney

TL;DR

This article presents an approach and architecture that empowers application developers to tune a model to their particular values, social norms, laws, and other regulations and orchestrate between potentially conflicting requirements in context.

Abstract

The alignment of large language models is usually done by model providers to add or control behaviors that are common or universally understood across use cases and contexts. In contrast, in this article, we present an approach and architecture that empowers application developers to tune a model to their particular values, social norms, laws and other regulations, and orchestrate between potentially conflicting requirements in context. We lay out three main components of such an Alignment Studio architecture: Framers, Instructors, and Auditors that work in concert to control the behavior of a language model. We illustrate this approach with a running example of aligning a company's internal-facing enterprise chatbot to its business conduct guidelines.

Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations

TL;DR

This article presents an approach and architecture that empowers application developers to tune a model to their particular values, social norms, laws, and other regulations and orchestrate between potentially conflicting requirements in context.

Abstract

The alignment of large language models is usually done by model providers to add or control behaviors that are common or universally understood across use cases and contexts. In contrast, in this article, we present an approach and architecture that empowers application developers to tune a model to their particular values, social norms, laws and other regulations, and orchestrate between potentially conflicting requirements in context. We lay out three main components of such an Alignment Studio architecture: Framers, Instructors, and Auditors that work in concert to control the behavior of a language model. We illustrate this approach with a running example of aligning a company's internal-facing enterprise chatbot to its business conduct guidelines.
Paper Structure (10 sections, 5 figures)

This paper contains 10 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Alignment Studio and Governance
  3. Framers
  4. Instructors
  5. Auditors
  6. Summary Demonstration
  7. Conclusion
  8. Acknowledgment

Figures (5)

  • Figure 1: A stylized depiction of Alignment Studio with its three components: Framers, Instructors, and Auditors.
  • Figure 2: A realization of the Alignment Studio software architecture, starting with policy documents. End-to-end software testing and documentation is recommended, but implementations need not include all components.
  • Figure 3: Creating instruction style seed data from policy documents (left) and using it to generate synthetic data using LLMs in a few-shot setting (right).
  • Figure 4: UI for comparing the responses of the unaligned and aligned model for a given prompt.
  • Figure 5: UI for evaluating responses for correctness and providing feedback on their quality.