Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pantagruel: Unified Self-Supervised Encoders for French Text and Speech

Phuong-Hang Le, Valentin Pelloin, Arnault Chatelain, Maryem Bouziane, Mohammed Ghennai, Qianwen Guan, Kirill Milintsevich, Salima Mdhaffar, Aidan Mannion, Nils Defauw, Shuyue Gu, Alexandre Audibert, Marco Dinarelli, Yannick Estève, Lorraine Goeuriot, Steffen Lalande, Nicolas Hervé, Maximin Coavoux, François Portet, Étienne Ollion, Marie Candito, Maxime Peyrard, Solange Rossato, Benjamin Lecouteux, Aurélie Nardy, Gilles Sérasset, Vincent Segonne, Solène Evain, Diandra Fabre, Didier Schwab

TL;DR

Pantagruel addresses unified self-supervised encoders for French text and speech by learning contextualized representations in feature space using a teacher–student framework (data2vec 2.0/JEPA) and augmenting text with a masked language modeling objective. It pre-trains text on Wikipedia, OSCAR, and CroissantLLM and speech on MLS, LeBenchmark, and a newly introduced INA-100k corpus of 100,000 hours of French audio. Across a broad suite of downstream tasks, Pantagruel matches or surpasses strong French baselines (CamemBERT, FlauBERT, LeBenchmark2.0) while maintaining a single architecture usable for both modalities, highlighting the effectiveness of feature-space objectives for French representation learning and multimodal understanding. The work shows that embedding-based targets work well for speech, while a hybrid embedding+MLM objective yields robust textual representations, with INA-100k improving robustness to acoustic variability and spontaneous speech; future directions include multi-modal training with unaligned data and scaling up models and corpora.

Abstract

We release Pantagruel models, a new family of self-supervised encoder models for French text and speech. Instead of predicting modality-tailored targets such as textual tokens or speech units, Pantagruel learns contextualized target representations in the feature space, allowing modality-specific encoders to capture linguistic and acoustic regularities more effectively. Separate models are pre-trained on large-scale French corpora, including Wikipedia, OSCAR and CroissantLLM for text, together with MultilingualLibriSpeech, LeBenchmark, and INA-100k for speech. INA-100k is a newly introduced 100,000-hour corpus of French audio derived from the archives of the Institut National de l'Audiovisuel (INA), the national repository of French radio and television broadcasts, providing highly diverse audio data. We evaluate Pantagruel across a broad range of downstream tasks spanning both modalities, including those from the standard French benchmarks such as FLUE or LeBenchmark. Across these tasks, Pantagruel models show competitive or superior performance compared to strong French baselines such as CamemBERT, FlauBERT, and LeBenchmark2.0, while maintaining a shared architecture that can seamlessly handle either speech or text inputs. These results confirm the effectiveness of feature-space self-supervised objectives for French representation learning and highlight Pantagruel as a robust foundation for multimodal speech-text understanding.

Pantagruel: Unified Self-Supervised Encoders for French Text and Speech

TL;DR

Pantagruel addresses unified self-supervised encoders for French text and speech by learning contextualized representations in feature space using a teacher–student framework (data2vec 2.0/JEPA) and augmenting text with a masked language modeling objective. It pre-trains text on Wikipedia, OSCAR, and CroissantLLM and speech on MLS, LeBenchmark, and a newly introduced INA-100k corpus of 100,000 hours of French audio. Across a broad suite of downstream tasks, Pantagruel matches or surpasses strong French baselines (CamemBERT, FlauBERT, LeBenchmark2.0) while maintaining a single architecture usable for both modalities, highlighting the effectiveness of feature-space objectives for French representation learning and multimodal understanding. The work shows that embedding-based targets work well for speech, while a hybrid embedding+MLM objective yields robust textual representations, with INA-100k improving robustness to acoustic variability and spontaneous speech; future directions include multi-modal training with unaligned data and scaling up models and corpora.

Abstract

We release Pantagruel models, a new family of self-supervised encoder models for French text and speech. Instead of predicting modality-tailored targets such as textual tokens or speech units, Pantagruel learns contextualized target representations in the feature space, allowing modality-specific encoders to capture linguistic and acoustic regularities more effectively. Separate models are pre-trained on large-scale French corpora, including Wikipedia, OSCAR and CroissantLLM for text, together with MultilingualLibriSpeech, LeBenchmark, and INA-100k for speech. INA-100k is a newly introduced 100,000-hour corpus of French audio derived from the archives of the Institut National de l'Audiovisuel (INA), the national repository of French radio and television broadcasts, providing highly diverse audio data. We evaluate Pantagruel across a broad range of downstream tasks spanning both modalities, including those from the standard French benchmarks such as FLUE or LeBenchmark. Across these tasks, Pantagruel models show competitive or superior performance compared to strong French baselines such as CamemBERT, FlauBERT, and LeBenchmark2.0, while maintaining a shared architecture that can seamlessly handle either speech or text inputs. These results confirm the effectiveness of feature-space self-supervised objectives for French representation learning and highlight Pantagruel as a robust foundation for multimodal speech-text understanding.
Paper Structure (35 sections, 1 figure, 10 tables)

This paper contains 35 sections, 1 figure, 10 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Models and Pre-training Framework
  5. Framework overview
  6. Loss functions
  7. Extension for better textual representations
  8. Pantagruel model configurations and implementation details
  9. Tokenizers
  10. Model configurations
  11. Implementation details
  12. Datasets and Resources
  13. Text datasets
  14. Speech datasets
  15. Ablation study
  16. ...and 20 more sections

Figures (1)

  • Figure 1: Overview of the Pantagruel model architecture. The network starts with a modality-specific pre-net to extract feature vectors from the input text/speech sequence. These features are input to a teacher encoder, while randomly chosen visible tokens (in blue) are input to a student encoder. A lightweight decoder predicts the teacher’s latent representations from the student’s outputs. For text input, an additional masked language modeling (MLM) loss is used. The teacher’s parameters are updated as an exponential moving average (EMA) of the student's. After training, only the embedding layer and the student encoder are used for fine-tuning on downstream tasks.