ReaCritic: Large Reasoning Transformer-based DRL Critic-model Scaling For Heterogeneous Networks

TL;DR

ReaCritic addresses the challenge of scalable, generalizable DRL in heterogeneous networks by introducing a large reasoning transformer-based critic. It combines horizontal reasoning across parallel state-action tokens with vertical transformer stacking to produce expressive Q-value estimates, and it is compatible with common DRL algorithms such as SAC. Empirical results show faster convergence, more stable learning, and better generalization across both HetNet resource-management tasks and standard OpenAI Gym controls, with ablations highlighting the value of inference-time-like reasoning and noise-induced diversity. The approach offers a practical path toward robust, scalable DRL for complex wireless environments where long-horizon dependencies and multi-objective tradeoffs are critical.

Abstract

Heterogeneous Networks (HetNets) pose critical challenges for intelligent management due to the diverse user requirements and time-varying wireless conditions. These factors introduce significant decision complexity, which limits the adaptability of existing Deep Reinforcement Learning (DRL) methods. In many DRL algorithms, especially those involving value-based or actor-critic structures, the critic component plays a key role in guiding policy learning by estimating value functions. However, conventional critic models often use shallow architectures that map observations directly to scalar estimates, limiting their ability to handle multi-task complexity. In contrast, recent progress in inference-time scaling of Large Language Models (LLMs) has shown that generating intermediate reasoning steps can significantly improve decision quality. Motivated by this, we propose ReaCritic, a large reasoning transformer-based criticmodel scaling scheme that brings reasoning ability into DRL. ReaCritic performs horizontal reasoning over parallel state-action inputs and vertical reasoning through deep transformer stacks. It is compatible with a broad range of value-based and actor-critic DRL algorithms and enhances generalization in dynamic wireless environments. Extensive experiments demonstrate that ReaCritic improves convergence speed and final performance across various HetNet settings and standard OpenAI Gym control tasks.

Figures (9)

• Figure 1: Comparison of DRL critic designs for HetNets. Conventional MLP-based critics (Part A) struggle with generalization in high-dimensional spaces. LLM-aided critics (Part B) improve expressiveness but face domain mismatch and integration challenges. Our proposed ReaCritic (Part C) introduces transformer-based bidirectional reasoning to achieve more accurate value estimation and scalable DRL training.
• Figure 2: System model of the proposed HetNet with one BS and $M$ heterogeneous users characterized by distinct communication demands, computational capabilities, and service requirement.
• Figure 3: The architecture of the proposed ReaCritic, which integrates two-dimensional reasoning: horizontal expansion via HRea steps to enrich token diversity, and vertical abstraction via VRea steps to enhance hierarchical value representation.
• Figure 4: Integration of ReaCritic into DRL algorithms. The part $1$ shows the reasoning-augmented SAC pipeline. Parts $2 \sim 5$ illustrate compatibility with other actor-critic methods, including TD3, A3C, DDPG, and PPO.
• Figure 5: Comparison of training performance with block number to be $4$, and user number to be 10 across ReaCritic-based SAC method with varying numbers of HRea steps and VRea steps, and standard SAC. The final reward ReaCritic-based SAC with different settings of HRea step and VRea step for 10 users is also included.