Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

Igor Jankowski

Abstract

While Multi-Agent Reinforcement Learning (MARL) algorithms achieve unprecedented successes across complex continuous domains, their standard deployment strictly adheres to a synchronous operational paradigm. Under this paradigm, agents are universally forced to execute deep neural network inferences at every micro-frame, regardless of immediate necessity. This dense throughput acts as a fundamental barrier to physical deployment on edge-devices where thermal and metabolic budgets are highly constrained. We propose Epistemic Time-Dilation MAPPO (ETD-MAPPO), augmented with a Dual-Gated Epistemic Trigger. Instead of depending on rigid frame-skipping (macro-actions), agents autonomously modulate their execution frequency by interpreting aleatoric uncertainty (via Shannon entropy of their policy) and epistemic uncertainty (via state-value divergence in a Twin-Critic architecture). To format this, we structure the environment as a Semi-Markov Decision Process (SMDP) and build the SMDP-Aligned Asynchronous Gradient Masking Critic to ensure proper credit assignment. Empirical findings demonstrate massive improvements (> 60% relative baseline acquisition leaps) over current temporal models. By assessing LBF, MPE, and the 115-dimensional state space of Google Research Football (GRF), ETD correctly prevented premature policy collapse. Remarkably, this unconstrained approach leads to emergent Temporal Role Specialization, reducing computational overhead by a statistically dominant 73.6% entirely during off-ball execution without deteriorating centralized task dominance.

Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL

Abstract

While Multi-Agent Reinforcement Learning (MARL) algorithms achieve unprecedented successes across complex continuous domains, their standard deployment strictly adheres to a synchronous operational paradigm. Under this paradigm, agents are universally forced to execute deep neural network inferences at every micro-frame, regardless of immediate necessity. This dense throughput acts as a fundamental barrier to physical deployment on edge-devices where thermal and metabolic budgets are highly constrained. We propose Epistemic Time-Dilation MAPPO (ETD-MAPPO), augmented with a Dual-Gated Epistemic Trigger. Instead of depending on rigid frame-skipping (macro-actions), agents autonomously modulate their execution frequency by interpreting aleatoric uncertainty (via Shannon entropy of their policy) and epistemic uncertainty (via state-value divergence in a Twin-Critic architecture). To format this, we structure the environment as a Semi-Markov Decision Process (SMDP) and build the SMDP-Aligned Asynchronous Gradient Masking Critic to ensure proper credit assignment. Empirical findings demonstrate massive improvements (> 60% relative baseline acquisition leaps) over current temporal models. By assessing LBF, MPE, and the 115-dimensional state space of Google Research Football (GRF), ETD correctly prevented premature policy collapse. Remarkably, this unconstrained approach leads to emergent Temporal Role Specialization, reducing computational overhead by a statistically dominant 73.6% entirely during off-ball execution without deteriorating centralized task dominance.
Paper Structure (23 sections, 7 equations, 5 figures, 4 tables, 2 algorithms)

This paper contains 23 sections, 7 equations, 5 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Compute-Aware Reasoning and Differentiable Execution
  4. Asynchronous and Temporal Hierarchy
  5. Epistemic Uncertainty via Ensemble Approximation
  6. Methodology
  7. The Dual-Gated Epistemic Execution Trigger
  8. Aleatoric Uncertainty via Policy Entropy
  9. Epistemic Uncertainty via Twin-Critic Divergence
  10. Deep Temporal Network Architecture
  11. SMDP-Aligned Asynchronous Integration
  12. Theoretical Error Bounds in SMDP
  13. Experimental Setup
  14. Environment Specifications
  15. Reproducibility and Hyperparameters
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Algorithmic Logic Flowchart. The environment loop executes exclusively against the strictly sequential evaluations of both Aleatoric inference outputs and True Epistemic variation.
  • Figure 2: Detailed Neural Architecture depicting exact hidden state preservation boundaries ($m_t$).
  • Figure 3: Empirical training progression across Level-Based Foraging. The Dual-Gated ETD model statistically outperforms fixed-parameter models natively driving explicit success logic.
  • Figure 4: Recorded 20,000 algorithmic updates modeling GRF acquisition states.
  • Figure 5: Mapping explicit continuous allocation ranges mapping MPE environments accurately targeting structural differences across agents.