Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model

Christopher Akiki, Giada Pistilli, Margot Mieskes, Matthias Gallé, Thomas Wolf, Suzana Ilić, Yacine Jernite

TL;DR

The paper analyzes how to conduct large-scale multilingual LLM research through a value-driven, participatory process that foregrounds governance, ethics, and data stewardship. It details BigScience's distributed organization, topic mapping into Working Groups, an ethical charter, and data governance practices that culminate in ROOTS and the BLOOM model, alongside a broad ecosystem of publications. It highlights outcomes such as a 1.6TB ROOTS corpus, open collaboration across 38 countries, and a suite of artifacts beyond BLOOM, while candidly addressing trade-offs like governance demands and funding constraints. Overall, the work offers a blueprint for responsible, inclusive AI research that can guide future participatory ML initiatives toward balancing openness with rigorous social and ethical considerations.

Abstract

The BigScience Workshop was a value-driven initiative that spanned one and half years of interdisciplinary research and culminated in the creation of ROOTS, a 1.6TB multilingual dataset that was used to train BLOOM, one of the largest multilingual language models to date. In addition to the technical outcomes and artifacts, the workshop fostered multidisciplinary collaborations around large models, datasets, and their analysis. This in turn led to a wide range of research publications spanning topics from ethics to law, data governance, modeling choices and distributed training. This paper focuses on the collaborative research aspects of BigScience and takes a step back to look at the challenges of large-scale participatory research, with respect to participant diversity and the tasks required to successfully carry out such a project. Our main goal is to share the lessons we learned from this experience, what we could have done better and what we did well. We show how the impact of such a social approach to scientific research goes well beyond the technical artifacts that were the basis of its inception.

BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model

TL;DR

The paper analyzes how to conduct large-scale multilingual LLM research through a value-driven, participatory process that foregrounds governance, ethics, and data stewardship. It details BigScience's distributed organization, topic mapping into Working Groups, an ethical charter, and data governance practices that culminate in ROOTS and the BLOOM model, alongside a broad ecosystem of publications. It highlights outcomes such as a 1.6TB ROOTS corpus, open collaboration across 38 countries, and a suite of artifacts beyond BLOOM, while candidly addressing trade-offs like governance demands and funding constraints. Overall, the work offers a blueprint for responsible, inclusive AI research that can guide future participatory ML initiatives toward balancing openness with rigorous social and ethical considerations.

Abstract

The BigScience Workshop was a value-driven initiative that spanned one and half years of interdisciplinary research and culminated in the creation of ROOTS, a 1.6TB multilingual dataset that was used to train BLOOM, one of the largest multilingual language models to date. In addition to the technical outcomes and artifacts, the workshop fostered multidisciplinary collaborations around large models, datasets, and their analysis. This in turn led to a wide range of research publications spanning topics from ethics to law, data governance, modeling choices and distributed training. This paper focuses on the collaborative research aspects of BigScience and takes a step back to look at the challenges of large-scale participatory research, with respect to participant diversity and the tasks required to successfully carry out such a project. Our main goal is to share the lessons we learned from this experience, what we could have done better and what we did well. We show how the impact of such a social approach to scientific research goes well beyond the technical artifacts that were the basis of its inception.
Paper Structure (12 sections, 2 figures)

This paper contains 12 sections, 2 figures.

Table of Contents

  1. BigScience Workshop---Context and Inception
  2. Value-Driven Science: Organization, Governance, and Participation
  3. Mapping Research Topics
  4. Distributed Project Organization, Governance, and Diversity
  5. Workshop Organization and Communication Mechanisms
  6. Aligning Goals through an Ethical Charter
  7. Building Diversity
  8. BigScience Participants Post Hoc Diversity and Feedback Survey
  9. Lessons Learned, Workshop Outputs, and the Future of BigScience
  10. Legal entity or ad-hoc collaboration.
  11. Breadth, time, and participation.
  12. Flexible goals and planning ahead.

Figures (2)

  • Figure 1: Geographic location of residence for 308 BigScience participants with a at least one traced contribution. This corresponds to 38 countries. (See Section \ref{['effective-diversity']} for more information.)
  • Figure 2: The BigScience working groups