Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family

Pierre-Carl Langlais, Pavel Chizhov, Mattia Nee, Carlos Rosas Hinostroza, Matthieu Delsart, Irène Girard, Othman Hicheur, Anastasia Stasenko, Ivan P. Yamshchikov

TL;DR

The paper tackles the challenge of reliable reasoning and source grounding in small language models used for retrieval-augmented generation. It introduces Pleias-RAG-350m and Pleias-RAG-1B, mid-trained on the open Common Corpus to achieve native citation, multilingual grounding, and robust RAG capabilities in constrained environments. The authors demonstrate Pareto-optimal performance in standard RAG benchmarks, competitive results with larger models, and strong cross-language grounding with minimal language-dependent degradation; they also propose architectural features like external memory, tokenizer recycling, and structured reasoning traces. They discuss deployment in regulated industries, with built-in traceability, governance, and an ethical framework, and outline future work including longer context and search integration.

Abstract

We introduce a new generation of small reasoning models for RAG, search, and source summarization. Pleias-RAG-350m and Pleias-RAG-1B are mid-trained on a large synthetic dataset emulating the retrieval of a wide variety of multilingual open sources from the Common Corpus. They provide native support for citation and grounding with literal quotes and reintegrate multiple features associated with RAG workflows, such as query routing, query reformulation, and source reranking. Pleias-RAG-350m and Pleias-RAG-1B outperform SLMs below 4 billion parameters on standardized RAG benchmarks (HotPotQA, 2wiki) and are competitive with popular larger models, including Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B. They are the only SLMs to date maintaining consistent RAG performance across leading European languages and ensuring systematic reference grounding for statements. Due to their size and ease of deployment on constrained infrastructure and higher factuality by design, the models unlock a range of new use cases for generative AI.

Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family

TL;DR

The paper tackles the challenge of reliable reasoning and source grounding in small language models used for retrieval-augmented generation. It introduces Pleias-RAG-350m and Pleias-RAG-1B, mid-trained on the open Common Corpus to achieve native citation, multilingual grounding, and robust RAG capabilities in constrained environments. The authors demonstrate Pareto-optimal performance in standard RAG benchmarks, competitive results with larger models, and strong cross-language grounding with minimal language-dependent degradation; they also propose architectural features like external memory, tokenizer recycling, and structured reasoning traces. They discuss deployment in regulated industries, with built-in traceability, governance, and an ethical framework, and outline future work including longer context and search integration.

Abstract

We introduce a new generation of small reasoning models for RAG, search, and source summarization. Pleias-RAG-350m and Pleias-RAG-1B are mid-trained on a large synthetic dataset emulating the retrieval of a wide variety of multilingual open sources from the Common Corpus. They provide native support for citation and grounding with literal quotes and reintegrate multiple features associated with RAG workflows, such as query routing, query reformulation, and source reranking. Pleias-RAG-350m and Pleias-RAG-1B outperform SLMs below 4 billion parameters on standardized RAG benchmarks (HotPotQA, 2wiki) and are competitive with popular larger models, including Qwen-2.5-7B, Llama-3.1-8B, and Gemma-3-4B. They are the only SLMs to date maintaining consistent RAG performance across leading European languages and ensuring systematic reference grounding for statements. Due to their size and ease of deployment on constrained infrastructure and higher factuality by design, the models unlock a range of new use cases for generative AI.

Paper Structure

This paper contains 24 sections, 9 figures.

Table of Contents

  1. Introduction
  2. Model design
  3. Grounding and verifiability
  4. Structured reasoning
  5. Tokenizer recycling
  6. Mid-training
  7. Definition
  8. Retrieval dataset
  9. Adversarial exercises
  10. Synthetic reasoning
  11. Training schedule
  12. Evaluation
  13. Standard benchmark
  14. Results
  15. Qualitative evaluation
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Scores on HotPotQA evaluation versus model size. Both Pleias models are Pareto-optimal among SLMs for RAG.
  • Figure 2: Comparison of a reconstruction of Anthropic citation mode vs. our method of straight generated citation, allowing for more direct intervention, such as citation shortening
  • Figure 3: Main scenarios incorporated into the reasoning model: trivial question (with a shortened reasoning mode), standard question, and refusal due to lack of source backing
  • Figure 4: Standardized RAG workflow integrated within the model also featuring further options available for implementation (query re-submission, standard refusal)
  • Figure 5: Token reassignment strategy for the RAG specialized models.
  • ...and 4 more figures