Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CADENT: Gated Hybrid Distillation for Sample-Efficient Transfer in Reinforcement Learning

Mahyar Alinejad, Yue Wang, George Atia

TL;DR

This work tackles sample-efficient transfer in reinforcement learning under domain shift by introducing CADENT, a framework that unifies long-horizon automaton-based strategic guidance with short-horizon policy-level tactics. It adds an experience-gated trust mechanism that dynamically arbitrates between the teacher's guidance and the student's own evolving experience at the state-action level, enabling graceful adaptation to target domains. The method combines a strategic intrinsic reward r_AD derived from automaton transitions with a tactical policy prior g_PD and gates their influence via a volatility-based trust omega(s,a). Empirically, CADENT achieves 40-60% better sample efficiency while maintaining strong asymptotic performance across diverse tasks, from sparse gridworlds to high-dimensional continuous control, establishing a robust paradigm for adaptive, multi-level knowledge transfer in RL.

Abstract

Transfer learning promises to reduce the high sample complexity of deep reinforcement learning (RL), yet existing methods struggle with domain shift between source and target environments. Policy distillation provides powerful tactical guidance but fails to transfer long-term strategic knowledge, while automaton-based methods capture task structure but lack fine-grained action guidance. This paper introduces Context-Aware Distillation with Experience-gated Transfer (CADENT), a framework that unifies strategic automaton-based knowledge with tactical policy-level knowledge into a coherent guidance signal. CADENT's key innovation is an experience-gated trust mechanism that dynamically weighs teacher guidance against the student's own experience at the state-action level, enabling graceful adaptation to target domain specifics. Across challenging environments, from sparse-reward grid worlds to continuous control tasks, CADENT achieves 40-60\% better sample efficiency than baselines while maintaining superior asymptotic performance, establishing a robust approach for adaptive knowledge transfer in RL.

CADENT: Gated Hybrid Distillation for Sample-Efficient Transfer in Reinforcement Learning

TL;DR

This work tackles sample-efficient transfer in reinforcement learning under domain shift by introducing CADENT, a framework that unifies long-horizon automaton-based strategic guidance with short-horizon policy-level tactics. It adds an experience-gated trust mechanism that dynamically arbitrates between the teacher's guidance and the student's own evolving experience at the state-action level, enabling graceful adaptation to target domains. The method combines a strategic intrinsic reward r_AD derived from automaton transitions with a tactical policy prior g_PD and gates their influence via a volatility-based trust omega(s,a). Empirically, CADENT achieves 40-60% better sample efficiency while maintaining strong asymptotic performance across diverse tasks, from sparse gridworlds to high-dimensional continuous control, establishing a robust paradigm for adaptive, multi-level knowledge transfer in RL.

Abstract

Transfer learning promises to reduce the high sample complexity of deep reinforcement learning (RL), yet existing methods struggle with domain shift between source and target environments. Policy distillation provides powerful tactical guidance but fails to transfer long-term strategic knowledge, while automaton-based methods capture task structure but lack fine-grained action guidance. This paper introduces Context-Aware Distillation with Experience-gated Transfer (CADENT), a framework that unifies strategic automaton-based knowledge with tactical policy-level knowledge into a coherent guidance signal. CADENT's key innovation is an experience-gated trust mechanism that dynamically weighs teacher guidance against the student's own experience at the state-action level, enabling graceful adaptation to target domain specifics. Across challenging environments, from sparse-reward grid worlds to continuous control tasks, CADENT achieves 40-60\% better sample efficiency than baselines while maintaining superior asymptotic performance, establishing a robust approach for adaptive knowledge transfer in RL.
Paper Structure (28 sections, 1 theorem, 11 equations, 8 figures, 1 algorithm)

This paper contains 28 sections, 1 theorem, 11 equations, 8 figures, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. Knowledge Transfer in Reinforcement Learning
  3. Policy-Level Transfer.
  4. Structure-Based Transfer.
  5. Adaptive Transfer Mechanisms.
  6. The Gap and Our Contribution
  7. PRELIMINARIES
  8. Policy Distillation
  9. Automaton-based Task Representation
  10. PROBLEM FORMULATION
  11. METHODOLOGY: THE CADENT FRAMEWORK
  12. Hybrid Distillation: Unifying Strategic and Tactical Guidance
  13. Strategic Guidance as Intrinsic Reward.
  14. Tactical Guidance as a Policy Prior.
  15. Experience-Gated Trust Mechanism
  16. ...and 13 more sections

Key Result

Proposition 6.1

In the tabular setting with bounded rewards $|R(s,a)| \leq R_{max}$ and bounded teacher guidance $|Q_{AD}(q,q')| \leq Q_{max}^{AD}$, the CADENT update satisfies This ensures updates remain bounded regardless of the trust mechanism's behavior.

Figures (8)

  • Figure 1: DFA for Blind Craftsman. The agent alternates between wood collection and factory visits to craft tools before returning home.
  • Figure 2: DFA for Dungeon Quest. Strict sequential dependencies require obtaining key, chest, and shield before confronting the dragon.
  • Figure 3: DFA for Mountain Car Collection. Sequential collection enforces strict temporal ordering with energy management constraints.
  • Figure 4: DFA for Warehouse Robotics. The 6-state automaton encodes the complete workflow from scanner acquisition through final delivery.
  • Figure 5: Reward per episode across all four environments. Top: Blind Craftsman (left), Dungeon Quest (right). Bottom: Mountain Car Collection (left), Warehouse Robotics (right).
  • ...and 3 more figures

Theorems & Definitions (2)

  • Proposition 6.1: Bounded Updates
  • proof : Proof Sketch