Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks

Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah

TL;DR

6G-Bench defines a taxonomy of 30 decision-making tasks extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories.

Abstract

This paper introduces 6G-Bench, an open benchmark for evaluating semantic communication and network-level reasoning in AI-native 6G networks. 6G-Bench defines a taxonomy of 30 decision-making tasks (T1--T30) extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories. Starting from 113,475 scenarios, we generate a balanced pool of 10,000 very-hard multiple-choice questions using task-conditioned prompts that enforce multi-step quantitative reasoning under uncertainty and worst-case regret minimization over multi-turn horizons. After automated filtering and expert human validation, 3,722 questions are retained as a high-confidence evaluation set, while the full pool is released to support training and fine-tuning of 6G-specialized models. Using 6G-Bench, we evaluate 22 foundation models spanning dense and mixture-of-experts architectures, short- and long-context designs (up to 1M tokens), and both open-weight and proprietary systems. Across models, deterministic single-shot accuracy (pass@1) spans a wide range from 0.22 to 0.82, highlighting substantial variation in semantic reasoning capability. Leading models achieve intent and policy reasoning accuracy in the range 0.87--0.89, while selective robustness analysis on reasoning-intensive tasks shows pass@5 values ranging from 0.20 to 0.91. To support open science and reproducibility, we release the 6G-Bench dataset on GitHub: https://github.com/maferrag/6G-Bench

6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks

TL;DR

6G-Bench defines a taxonomy of 30 decision-making tasks extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories.

Abstract

This paper introduces 6G-Bench, an open benchmark for evaluating semantic communication and network-level reasoning in AI-native 6G networks. 6G-Bench defines a taxonomy of 30 decision-making tasks (T1--T30) extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories. Starting from 113,475 scenarios, we generate a balanced pool of 10,000 very-hard multiple-choice questions using task-conditioned prompts that enforce multi-step quantitative reasoning under uncertainty and worst-case regret minimization over multi-turn horizons. After automated filtering and expert human validation, 3,722 questions are retained as a high-confidence evaluation set, while the full pool is released to support training and fine-tuning of 6G-specialized models. Using 6G-Bench, we evaluate 22 foundation models spanning dense and mixture-of-experts architectures, short- and long-context designs (up to 1M tokens), and both open-weight and proprietary systems. Across models, deterministic single-shot accuracy (pass@1) spans a wide range from 0.22 to 0.82, highlighting substantial variation in semantic reasoning capability. Leading models achieve intent and policy reasoning accuracy in the range 0.87--0.89, while selective robustness analysis on reasoning-intensive tasks shows pass@5 values ranging from 0.20 to 0.91. To support open science and reproducibility, we release the 6G-Bench dataset on GitHub: https://github.com/maferrag/6G-Bench
Paper Structure (46 sections, 17 equations, 5 figures, 10 tables, 2 algorithms)

This paper contains 46 sections, 17 equations, 5 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Telecommunications Knowledge Benchmarks
  4. Reasoning over Telecommunications Standards and Tables
  5. Agentic Evaluation in Telecommunication Service Scenarios
  6. Reasoning and Optimization in Wireless Communications
  7. Security-Constrained Wireless Communication Benchmarks
  8. Positioning of 6G-Bench
  9. Benchmark Design and Evaluation Methodology
  10. Problem Formulation and Episode Source
  11. Episode Source
  12. Semantic Network State
  13. Semantic Communication Model
  14. Decision Space and State Evolution
  15. Objective: Network-Level Semantic Reasoning
  16. ...and 31 more sections

Figures (5)

  • Figure 1: Overview of the 6G-Bench benchmark design, task construction, validation, and evaluation pipeline.
  • Figure 2: Network condition diversity in 6G-Bench. Subfigures (a)--(c) show the proportion of MCQ questions operating under different latency, throughput, and edge-compute load regimes.
  • Figure 3: Distribution of per-question maximum network metrics in 6G-Bench, including latency, throughput, edge-compute load, and packet loss.
  • Figure 4: Coverage of 6G-Bench across tasks and capability categories. Subfigure \ref{['fig:tasks-per-category']} shows that questions are distributed across the five standardization-aligned capability categories, while Subfigure \ref{['fig:tasks-per-task']} details the number of MCQ instances for each individual task T1--T30.
  • Figure 5: 6G-Bench pass@1 accuracy by model size category. Each subfigure shows pass@1 accuracy for models within a single parameter scale: (a) Large, (b) Medium, (c) Small, and (d) Tiny.