TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

TL;DR

This work addresses the gap in telecom-domain knowledge within general LLMs by proposing TelecomGPT, a three-stage pipeline (continual pre-training, instruct tuning, alignment tuning) built on three datasets (OpenTelecom, TelecomInstruct, TelecomAlign). It introduces a comprehensive telecom evaluation suite (TeleQnA, 3GPP WG classification, Telecom Math Modeling, Telecom Code tasks, and Instruct Following) to quantify capabilities in math modeling, code generation, and standards comprehension. Empirical results show TelecomGPT outperforms baselines on math modeling and achieves competitive performance on other benchmarks, highlighting the practicality of adapting general LLMs to domain-specific telecom tasks with cost-effective training. The study demonstrates a clear path toward telecom-enabled LLMs, with future work pointing to multimodal data integration and broader standard-domain coverage for even stronger deployment potential.

Abstract

Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the first time, we propose a pipeline to adapt any general purpose LLMs to a telecom-specific LLMs. We collect and build telecom-specific pre-train dataset, instruction dataset, preference dataset to perform continual pre-training, instruct tuning and alignment tuning respectively. Besides, due to the lack of widely accepted evaluation benchmarks in telecom domain, we extend existing evaluation benchmarks and proposed three new benchmarks, namely, Telecom Math Modeling, Telecom Open QnA and Telecom Code Tasks. These new benchmarks provide a holistic evaluation of the capabilities of LLMs including math modeling, Open-Ended question answering, code generation, infilling, summarization and analysis in telecom domain. Our fine-tuned LLM TelecomGPT outperforms state of the art (SOTA) LLMs including GPT-4, Llama-3 and Mistral in Telecom Math Modeling benchmark significantly and achieve comparable performance in various evaluation benchmarks such as TeleQnA, 3GPP technical documents classification, telecom code summary and generation and infilling.

Figures (8)

• Figure 1: The training pipeline of our TelecomGPT framework. The full pipeline consist of three training stage, namely, continual pretraining on telecom domain, instruct tuning (sft) and alignment tuning.
• Figure 2: Prompt template for creating mcq. Clear instructions are given to llm to avoid references.
• Figure 3: Prompt template for creating code generation instruction. Clear instructions and examples are given to llm to filter telecom-irrelevant functions or avoid generating meaningless request.
• Figure 4: Prompt template for creating task completion or planning. Clear instructions are given to llm to avoid references, implicit contents and guide llm to generate sequential steps.
• Figure 5: Training and evaluation loss during continue pretraining (LlaMA2-7B-TP).