Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Intrinsic Surprise-Regularized Control (DISRC): A Biologically Inspired Mechanism for Efficient Deep Q-Learning in Sparse Environments

Yash Kini, Shiv Davay, Shreya Polavarapu

TL;DR

The paper tackles the challenge of poor sample efficiency and unstable learning for Deep Q-Networks in sparse-reward settings by introducing Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired mechanism that modulates learning updates using an internal surprise signal. DISRC employs a LayerNorm-based encoder to map observations into a 64-dimensional latent space and computes a deviation-based surprise against a moving latent setpoint μ_t, scaling each TD update by both the TD error and the surprise intensity via a decaying plasticity factor β_t and a normalized reward adjustment hat r_t = r_t/EMA(|r|) + λ b_t. Empirical results on MiniGrid DoorKey-8x8 and LavaCrossing tasks show improved efficiency and stability, with faster first-success times and higher AUC in DoorKey (79 vs. 118 episodes; 0.25 vs. 0.34 reward SD; AUC 596.42 vs. 534.90) and higher final rewards and AUC in LavaCrossing (0.95 vs. 0.93; AUC 957.04 vs. 934.82), albeit with slower early convergence in some cases. These findings establish DISRC as a promising, neuro-inspired approach to regulate learning intensity in off-policy DRL, particularly in environments with sparse and delayed feedback.

Abstract

Deep reinforcement learning (DRL) has driven major advances in autonomous control. Still, standard Deep Q-Network (DQN) agents tend to rely on fixed learning rates and uniform update scaling, even as updates are modulated by temporal-difference (TD) error. This rigidity destabilizes convergence, especially in sparse-reward settings where feedback is infrequent. We introduce Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired augmentation to DQN that dynamically scales Q-updates based on latent-space surprise. DISRC encodes states via a LayerNorm-based encoder and computes a deviation-based surprise score relative to a moving latent setpoint. Each update is then scaled in proportion to both TD error and surprise intensity, promoting plasticity during early exploration and stability as familiarity increases. We evaluate DISRC on two sparse-reward MiniGrid environments, which included MiniGrid-DoorKey-8x8 and MiniGrid-LavaCrossingS9N1, under identical settings as a vanilla DQN baseline. In DoorKey, DISRC reached the first successful episode (reward > 0.8) 33% faster than the vanilla DQN baseline (79 vs. 118 episodes), with lower reward standard deviation (0.25 vs. 0.34) and higher reward area under the curve (AUC: 596.42 vs. 534.90). These metrics reflect faster, more consistent learning - critical for sparse, delayed reward settings. In LavaCrossing, DISRC achieved a higher final reward (0.95 vs. 0.93) and the highest AUC of all agents (957.04), though it converged more gradually. These preliminary results establish DISRC as a novel mechanism for regulating learning intensity in off-policy agents, improving both efficiency and stability in sparse-reward domains. By treating surprise as an intrinsic learning signal, DISRC enables agents to modulate updates based on expectation violations, enhancing decision quality when conventional value-based methods fall short.

Deep Intrinsic Surprise-Regularized Control (DISRC): A Biologically Inspired Mechanism for Efficient Deep Q-Learning in Sparse Environments

TL;DR

The paper tackles the challenge of poor sample efficiency and unstable learning for Deep Q-Networks in sparse-reward settings by introducing Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired mechanism that modulates learning updates using an internal surprise signal. DISRC employs a LayerNorm-based encoder to map observations into a 64-dimensional latent space and computes a deviation-based surprise against a moving latent setpoint μ_t, scaling each TD update by both the TD error and the surprise intensity via a decaying plasticity factor β_t and a normalized reward adjustment hat r_t = r_t/EMA(|r|) + λ b_t. Empirical results on MiniGrid DoorKey-8x8 and LavaCrossing tasks show improved efficiency and stability, with faster first-success times and higher AUC in DoorKey (79 vs. 118 episodes; 0.25 vs. 0.34 reward SD; AUC 596.42 vs. 534.90) and higher final rewards and AUC in LavaCrossing (0.95 vs. 0.93; AUC 957.04 vs. 934.82), albeit with slower early convergence in some cases. These findings establish DISRC as a promising, neuro-inspired approach to regulate learning intensity in off-policy DRL, particularly in environments with sparse and delayed feedback.

Abstract

Deep reinforcement learning (DRL) has driven major advances in autonomous control. Still, standard Deep Q-Network (DQN) agents tend to rely on fixed learning rates and uniform update scaling, even as updates are modulated by temporal-difference (TD) error. This rigidity destabilizes convergence, especially in sparse-reward settings where feedback is infrequent. We introduce Deep Intrinsic Surprise-Regularized Control (DISRC), a biologically inspired augmentation to DQN that dynamically scales Q-updates based on latent-space surprise. DISRC encodes states via a LayerNorm-based encoder and computes a deviation-based surprise score relative to a moving latent setpoint. Each update is then scaled in proportion to both TD error and surprise intensity, promoting plasticity during early exploration and stability as familiarity increases. We evaluate DISRC on two sparse-reward MiniGrid environments, which included MiniGrid-DoorKey-8x8 and MiniGrid-LavaCrossingS9N1, under identical settings as a vanilla DQN baseline. In DoorKey, DISRC reached the first successful episode (reward > 0.8) 33% faster than the vanilla DQN baseline (79 vs. 118 episodes), with lower reward standard deviation (0.25 vs. 0.34) and higher reward area under the curve (AUC: 596.42 vs. 534.90). These metrics reflect faster, more consistent learning - critical for sparse, delayed reward settings. In LavaCrossing, DISRC achieved a higher final reward (0.95 vs. 0.93) and the highest AUC of all agents (957.04), though it converged more gradually. These preliminary results establish DISRC as a novel mechanism for regulating learning intensity in off-policy agents, improving both efficiency and stability in sparse-reward domains. By treating surprise as an intrinsic learning signal, DISRC enables agents to modulate updates based on expectation violations, enhancing decision quality when conventional value-based methods fall short.
Paper Structure (12 sections, 1 equation, 3 figures, 1 table)

This paper contains 12 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Environments
  4. Baseline DQN Implementation
  5. DISRC Architecture
  6. State Encoder
  7. Surprise Controller
  8. Training Procedure
  9. Evaluation Metrics
  10. Code Availability
  11. Results
  12. Discussion

Figures (3)

  • Figure 1: DISRC Algorithm Pseudocode. This diagram outlines the core loop of Deep Intrinsic Surprise-Regularized Control (DISRC). The agent dynamically adjusts Q-value update magnitudes using a biologically inspired surprise score computed from deviation relative to a moving latent setpoint.
  • Figure 2: Learning Curves on DoorKey-8x8. DISRC demonstrates more rapid convergence and smoother learning signals than baseline DQN, along with tighter loss distributions.
  • Figure 3: Learning Curves on LavaCrossingS9N1. DISRC exhibits greater reward consistency and training stability under sparse, high-risk transitions.