Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COLE: a Comprehensive Benchmark for French Language Understanding Evaluation

David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, Richard Khoury

TL;DR

COLE presents a 23-task French NLU benchmark evaluated across 95 LLMs to map the current landscape of French language understanding. It emphasizes three task groups (single-sentence, similarity/paraphrase, inference) and uses a GLUE-style composite score to enable cross-task comparisons, highlighting a pronounced divide between closed- and open-weight models. Key findings show strong semantic and grammatical capabilities but persistent gaps in zero-shot extractive QA, regional language variation, and fine-grained word sense disambiguation, with dialectal data (Québec French) driving notable performance gaps. The work provides a public resource to drive progress in French LLMs, while acknowledging limitations such as data contamination, reliance on translations, and the need for dialect-specific analyses to ensure robust, fair evaluation across French varieties.

Abstract

To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.

COLE: a Comprehensive Benchmark for French Language Understanding Evaluation

TL;DR

COLE presents a 23-task French NLU benchmark evaluated across 95 LLMs to map the current landscape of French language understanding. It emphasizes three task groups (single-sentence, similarity/paraphrase, inference) and uses a GLUE-style composite score to enable cross-task comparisons, highlighting a pronounced divide between closed- and open-weight models. Key findings show strong semantic and grammatical capabilities but persistent gaps in zero-shot extractive QA, regional language variation, and fine-grained word sense disambiguation, with dialectal data (Québec French) driving notable performance gaps. The work provides a public resource to drive progress in French LLMs, while acknowledging limitations such as data contamination, reliance on translations, and the need for dialect-specific analyses to ensure robust, fair evaluation across French varieties.

Abstract

To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.

Paper Structure

This paper contains 41 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. COLE Benchmark
  4. Tasks
  5. Single-Sentence Tasks
  6. Similarity and Paraphrase Tasks
  7. Inference Tasks
  8. Evaluation Metrics
  9. Task Evaluations
  10. Composite Score
  11. Experiments
  12. Evaluation Settings
  13. Models
  14. Baseline Models
  15. LLM
  16. ...and 26 more sections

Figures (2)

  • Figure 1: Composite score performance of all 95 LLMs. Scores are reported as percentages (%) and are ranked in descending order ($\blacktriangledown$).
  • Figure 2: The translated prompt templates used for the zero-shot evaluation of each task in the COLE benchmark. Each prompt consists of a system message providing the instruction and a user message containing the input placeholder for the data instance. Blue boxes contain the task instructions. Yellow boxes contain the prefix for the model to continue. Texts in "$\ll\gg$" are role-tags to be fed to the model.