An Autonomous Network Orchestration Framework Integrating Large Language Models with Continual Reinforcement Learning

TL;DR

This work tackles the challenging problem of resource orchestration in 6G Space-Air-Ground Integrated Networks (SAGIN) with Semantic Communication (SemCom), where traditional optimization is hampered by combinatorial complexity, semantic QoE evaluation, and unknown dynamics. It proposes Autonomous Reinforcement Coordination (ARC), a two-tier framework that delegates high-level planning to Large Language Models (LLMs) via a Retrieval-Augmented Generator (RAG) and low-level decision-making to a Mixture of Experts-inspired set of Reinforcement Learning (RL) agents managed by a Hierarchical Action Planner (HAP). ARC leverages Chain-of-Thought reasoning for few-shot learning, contrastive learning to refine exemplars, and replay-buffer-based continual learning to adapt to dynamic environments, thereby reducing hallucinations and improving robustness. The paper demonstrates ARC through simulations and discusses future directions, such as prediction-based state indexing, online LLM training, and Algorithm-of-Thought approaches, to further enhance performance and scalability in semantic-aware network orchestration.

Abstract

6G networks aim to achieve global coverage, massive connectivity, and ultra-stringent requirements. Space-Air-Ground Integrated Networks (SAGINs) and Semantic Communication (SemCom) are essential for realizing these goals, yet they introduce considerable complexity in resource orchestration. Drawing inspiration from research in robotics, a viable solution to manage this complexity is the application of Large Language Models (LLMs). Although the use of LLMs in network orchestration has recently gained attention, existing solutions have not sufficiently addressed LLM hallucinations or their adaptation to network dynamics. To address this gap, this paper proposes a framework called Autonomous Reinforcement Coordination (ARC) for a SemCom-enabled SAGIN. This framework employs an LLM-based Retrieval-Augmented Generator (RAG) monitors services, users, and resources and processes the collected data, while a Hierarchical Action Planner (HAP) orchestrates resources. ARC decomposes orchestration into two tiers, utilizing LLMs for high-level planning and Reinforcement Learning (RL) agents for low-level decision-making, in alignment with the Mixture of Experts (MoE) concept. The LLMs utilize Chain-of-Thought (CoT) reasoning for few-shot learning, empowered by contrastive learning, while the RL agents employ replay buffer management for continual learning, thereby achieving efficiency, accuracy, and adaptability. Simulations are provided to demonstrate the effectiveness of ARC, along with a comprehensive discussion on potential future research directions to enhance and upgrade ARC.

Figures (4)

• Figure 1: The components of ARC.
• Figure 2: The resource allocation workflow in ARC. Note that the color of each sub-component and functionality follows its parent component in Fig. \ref{['fig1']}.
• Figure 3: The process of updating rewards in ARC to facilitate reward-based few-shot learning and gradient-based continual learning.
• Figure 4: Two scenarios comparing the results of ARC with optimal, A) reward-unaware, and B) non-reinforcement results. The normalized cost of each allocation is calculated to ensure that allocations with lower costs receive higher scores, and vice versa.