Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ministral 3

Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Clémence Lanfranchi, Corentin Barreau, Cyprien Courtot, Daniele Grattarola, Darius Dabert, Diego de las Casas, Elliot Chane-Sane, Faruk Ahmed, Gabrielle Berrada, Gaëtan Ecrepont, Gauthier Guinet, Georgii Novikov, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Gunshi Gupta, Jan Ludziejewski, Jason Rute, Joachim Studnia, Jonas Amar, Joséphine Delas, Josselin Somerville Roberts, Karmesh Yadav, Khyathi Chandu, Kush Jain, Laurence Aitchison, Laurent Fainsin, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Margaret Jennings, Marie Pellat, Mark Prins, Mathieu Poirée, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mia Chiquier, Michel Schimpf, Nathan Grinsztajn, Neha Gupta, Nikhil Raghuraman, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Patrick von Platen, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Quentin Torroba, Romain Sauvestre, Roman Soletskyi, Rupert Menneer, Sagar Vaze, Samuel Barry, Sanchit Gandhi, Siddhant Waghjale, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Théo Cachet, Theo Simon Sorg, Thibaut Lavril, Thiziri Nait Saada, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Bewley, Tom Edwards, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Van Phung, Vincent Maladière, Virgile Richard, Wassim Bouaziz, Wen-Ding Li, William Marshall, Xinghui Li, Xinyu Yang, Yassine El Ouahidi, Yihan Wang, Yunhao Tang, Zaccharie Ramzi

TL;DR

Ministral 3 presents a family of open-weight, compute-efficient dense language models (3B, 8B, 14B) derived from Mistral Small 3.1 using Cascade Distillation to prune and distill knowledge into smaller students while achieving long-context and multimodal capabilities. Each size is released in base, instruct, and reasoning variants, with vision encoders frozen during post-training and context lengths up to $256k$ tokens; post-training employs SFT, ODPO, and GRPO to align with instruction-following and complex reasoning tasks. The paper reports strong pretraining results against open baselines and competitive post-training performance across a wide range of benchmarks, and discusses critical design choices such as teacher selection, verbosity, and online preference optimization. Overall, Ministral 3 demonstrates that cascade distillation can yield high-performing, resource-efficient models suitable for constrained compute and memory environments and open-source deployment.

Abstract

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.

Ministral 3

TL;DR

Ministral 3 presents a family of open-weight, compute-efficient dense language models (3B, 8B, 14B) derived from Mistral Small 3.1 using Cascade Distillation to prune and distill knowledge into smaller students while achieving long-context and multimodal capabilities. Each size is released in base, instruct, and reasoning variants, with vision encoders frozen during post-training and context lengths up to tokens; post-training employs SFT, ODPO, and GRPO to align with instruction-following and complex reasoning tasks. The paper reports strong pretraining results against open baselines and competitive post-training performance across a wide range of benchmarks, and discusses critical design choices such as teacher selection, verbosity, and online preference optimization. Overall, Ministral 3 demonstrates that cascade distillation can yield high-performing, resource-efficient models suitable for constrained compute and memory environments and open-source deployment.

Abstract

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.
Paper Structure (22 sections, 1 equation, 6 figures, 5 tables, 2 algorithms)

This paper contains 22 sections, 1 equation, 6 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Training Recipe
  4. Pretraining
  5. Post-Training: Ministral Instruct
  6. Supervised Fine-tuning
  7. Online Direct Preference Optimization stage
  8. Post-Training: Ministral Reasoning
  9. Reasoning Supervised Fine-Tuning
  10. Reinforcement Learning
  11. Online Direct Preference Optimization
  12. Results
  13. Pretraining Results
  14. Post-training Results
  15. Discussions
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of Ministral 3 training recipe.Pretraining: We start from pruning the parent model, Mistral Small 3.1, into the largest child model (14B Init.). Next, we continue pretraining the child model with logit distillation from the parent model as the teacher to obtain the up-trained short context child model (14B Short Ctx.). From 14B Short Ctx., we perform another round of distillation with longer context window (see §\ref{['subsec:pt']} for details) to obtain the final Ministral 3 14B Base model. In parallel, 14B Short Ctx. is pruned to initialize the next child model (8B Init.), from which we repeat the process to derive Ministral 3 8B Base model. We repeat the same process for the 3B version. Post-training: Each Base model is then post-trained into the instruction-following and reasoning variants. For instruction-following, our post-training recipe includes supervised fine-tuning (SFT) and Online Direct Preference Optimization (ODPO). For reasoning, the process involved supervised fine-tuning with chain-of-thought data (SFT w/ CoT), Group Relative Policy Optimization (GRPO; shao2024deepseekmathpushinglimitsmathematical), and ODPO.
  • Figure 2: Cascade Distillation.
  • Figure 3: Ministral 3 14B pretraining ablations comparing distillation from Mistral Small 3.1 and Mistral Medium 3 teachers. Despite Mistral Medium 3 being larger and more capable, distillation from Mistral Small 3.1 consistently yields stronger downstream performance across different benchmarks.
  • Figure 4: Ministral 3 3B pretraining ablations comparing distillation from base and post-trained (instruct/reasoning) variants of Mistral Small 3.1. The instruct teacher yields stronger performance on STEM benchmarks, while achieving comparable results on knowledge and multimodal evaluations..
  • Figure 5: Verbosity (in terms of number of output tokens) v.s. accuracy on GPQA Diamond with Ministral 3 instruction-following and reasoning.
  • ...and 1 more figures