Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, Yulia Tsvetkov

TL;DR

Modular Pluralism is a modular framework based on multi-LLM collaboration for pluralistic alignment that “plugs into” a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional.

Abstract

While existing alignment paradigms have been integral in developing large language models (LLMs), LLMs often learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities. We propose Modular Pluralism, a modular framework based on multi-LLM collaboration for pluralistic alignment: it "plugs into" a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional. Modular Pluralism is uniquely compatible with black-box LLMs and offers the modular control of adding new community LMs for previously underrepresented communities. We evaluate Modular Pluralism with six tasks and four datasets featuring questions/instructions with value-laden and perspective-informed responses. Extensive experiments demonstrate that Modular Pluralism advances the three pluralism objectives across six black-box and open-source LLMs. Further analysis reveals that LLMs are generally faithful to the inputs from smaller community LLMs, allowing seamless patching by adding a new community LM to better cover previously underrepresented communities.

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

TL;DR

Modular Pluralism is a modular framework based on multi-LLM collaboration for pluralistic alignment that “plugs into” a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional.

Abstract

While existing alignment paradigms have been integral in developing large language models (LLMs), LLMs often learn an averaged human preference and struggle to model diverse preferences across cultures, demographics, and communities. We propose Modular Pluralism, a modular framework based on multi-LLM collaboration for pluralistic alignment: it "plugs into" a base LLM a pool of smaller but specialized community LMs, where models collaborate in distinct modes to flexibility support three modes of pluralism: Overton, steerable, and distributional. Modular Pluralism is uniquely compatible with black-box LLMs and offers the modular control of adding new community LMs for previously underrepresented communities. We evaluate Modular Pluralism with six tasks and four datasets featuring questions/instructions with value-laden and perspective-informed responses. Extensive experiments demonstrate that Modular Pluralism advances the three pluralism objectives across six black-box and open-source LLMs. Further analysis reveals that LLMs are generally faithful to the inputs from smaller community LLMs, allowing seamless patching by adding a new community LM to better cover previously underrepresented communities.
Paper Structure (36 sections, 10 figures, 10 tables)

This paper contains 36 sections, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Modular Framework
  4. Overton Pluralism
  5. Steerable Pluralism
  6. Distributional Pluralism
  7. Experimental Settings
  8. Models
  9. Implementation
  10. Baselines
  11. Tasks and Datasets
  12. Results
  13. Modular Pluralism better covers diverse values and perspectives.
  14. Human and GPT-4 evaluation find that Modular Pluralism produces more pluralistic responses.
  15. Modular Pluralism offers stronger steerability for value-specific contexts.
  16. ...and 21 more sections

Figures (10)

  • Figure 1: Overview of Modular Pluralism, where a large language model interact with a pool of smaller but specialized community LMs for pluralistic alignment. Depending on the three pluralistic alignment objectives, the LLM either functions as a multi-document summarization system, selects the most fitting community, or produces aggregated distributions separately conditioned on each community LM's comments.
  • Figure 2: Results for Overton w/ NLI evaluation. Modular Pluralism with the aligned LLM successfully improves value coverage against the strongest baseline by 27.8% and 50.3% for the two LLMs.
  • Figure 3: Results for Overton w/ human and GPT-4 evaluation with the ChatGPT LLM. Modular Pluralism has a 16.5% and 45.8% higher win rate against the strongest baseline.
  • Figure 4: Results for Distributional w/ MoralChoice in Jensen-Shannon distance, the lower the better. While unaligned and unaligned models show distinctly different patterns in low and high-ambiguity moral scenarios, Modular Pluralism consistently improves over baselines in overall distributional distances.
  • Figure 5: Coverage percentages of the community LMs' comments in the LLM's final response, and the percentage of new content added by the LLM: the higher the better. We find moderate coverage of 40% to 60% for community LM comments, while 20% to 40% sentences in the final response are new content added by the LLM.
  • ...and 5 more figures