Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CovertComBench: The First Domain-Specific Testbed for LLMs in Wireless Covert Communication

Zhaozhi Liu, Jiaxin Chen, Yuanai Xie, Yuna Jiang, Minrui Xu, Xiao Zhang, Pan Lai, Zan Zhou

TL;DR

This work introduces CovertComBench, the first domain-specific benchmark for evaluating LLMs in wireless covert communication under strict covertness constraints. The benchmark comprises MCQs, ODQs, and CGQs to probe conceptual understanding, symbolic optimization, and code generation within CC scenarios, formalized by a detection constraint such as $\mathrm{KL}(P_w \| P_a) \le \varepsilon$. An evaluation framework combines human expert and LLM-as-Judge assessments to measure reliability and bias in automated scoring. Empirical results show that LLMs excel at MCQs and CGQs but struggle with multi-step optimization tasks, underscoring the need for tool augmentation and closed-loop reasoning to build trustworthy, domain-specific wireless AI systems.

Abstract

The integration of Large Language Models (LLMs) into wireless networks presents significant potential for automating system design. However, unlike conventional throughput maximization, Covert Communication (CC) requires optimizing transmission utility under strict detection-theoretic constraints, such as Kullback-Leibler divergence limits. Existing benchmarks primarily focus on general reasoning or standard communication tasks and do not adequately evaluate the ability of LLMs to satisfy these rigorous security constraints. To address this limitation, we introduce CovertComBench, a unified benchmark designed to assess LLM capabilities across the CC pipeline, encompassing conceptual understanding (MCQs), optimization derivation (ODQs), and code generation (CGQs). Furthermore, we analyze the reliability of automated scoring within a detection-theoretic ``LLM-as-Judge'' framework. Extensive evaluations across state-of-the-art models reveal a significant performance discrepancy. While LLMs achieve high accuracy in conceptual identification (81%) and code implementation (83%), their performance in the higher-order mathematical derivations necessary for security guarantees ranges between 18% and 55%. This limitation indicates that current LLMs serve better as implementation assistants rather than autonomous solvers for security-constrained optimization. These findings suggest that future research should focus on external tool augmentation to build trustworthy wireless AI systems.

CovertComBench: The First Domain-Specific Testbed for LLMs in Wireless Covert Communication

TL;DR

This work introduces CovertComBench, the first domain-specific benchmark for evaluating LLMs in wireless covert communication under strict covertness constraints. The benchmark comprises MCQs, ODQs, and CGQs to probe conceptual understanding, symbolic optimization, and code generation within CC scenarios, formalized by a detection constraint such as . An evaluation framework combines human expert and LLM-as-Judge assessments to measure reliability and bias in automated scoring. Empirical results show that LLMs excel at MCQs and CGQs but struggle with multi-step optimization tasks, underscoring the need for tool augmentation and closed-loop reasoning to build trustworthy, domain-specific wireless AI systems.

Abstract

The integration of Large Language Models (LLMs) into wireless networks presents significant potential for automating system design. However, unlike conventional throughput maximization, Covert Communication (CC) requires optimizing transmission utility under strict detection-theoretic constraints, such as Kullback-Leibler divergence limits. Existing benchmarks primarily focus on general reasoning or standard communication tasks and do not adequately evaluate the ability of LLMs to satisfy these rigorous security constraints. To address this limitation, we introduce CovertComBench, a unified benchmark designed to assess LLM capabilities across the CC pipeline, encompassing conceptual understanding (MCQs), optimization derivation (ODQs), and code generation (CGQs). Furthermore, we analyze the reliability of automated scoring within a detection-theoretic ``LLM-as-Judge'' framework. Extensive evaluations across state-of-the-art models reveal a significant performance discrepancy. While LLMs achieve high accuracy in conceptual identification (81%) and code implementation (83%), their performance in the higher-order mathematical derivations necessary for security guarantees ranges between 18% and 55%. This limitation indicates that current LLMs serve better as implementation assistants rather than autonomous solvers for security-constrained optimization. These findings suggest that future research should focus on external tool augmentation to build trustworthy wireless AI systems.
Paper Structure (12 sections, 4 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 12 sections, 4 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The CovertComBench
  3. Motivation
  4. CovertComBench Construction Pipeline
  5. Question Structure and Design
  6. Dataset Statistics and Splits
  7. Experiments
  8. Experiment Setup
  9. Evaluation Metrics
  10. Experiment Result
  11. Main Error Analysis
  12. Conclusion

Figures (2)

  • Figure 1: The construction pipeline of CovertComBench. The four main stages are: (1) Contamination Check, (2) Context Extraction, (3) Question Construction, and (4) Experts Review.
  • Figure 2: CovertComBench difficulty distribution.