Enhanced-FQL($λ$), an Efficient and Interpretable RL with novel Fuzzy Eligibility Traces and Segmented Experience Replay

TL;DR

The paper addresses the limitations of deep RL in continuous control by introducing Enhanced-FQL($\lambda$), a fuzzy reinforcement learning framework that uses a fuzzified Bellman operator with eligibility traces and segmented experience replay. By replacing neural networks with an interpretable fuzzy rule base, it achieves stable multi-step credit assignment, improved sample efficiency, and lower computational cost. The authors prove convergence under standard assumptions and demonstrate faster learning and reduced variance compared with baselines like n-step fuzzy TD and fuzzy SARSA($\lambda$), while approaching the performance of DDPG with significantly less computation. This approach is particularly suitable for safety-critical, resource-constrained scenarios where transparency and reliability are essential.

Abstract

This paper introduces a fuzzy reinforcement learning framework, Enhanced-FQL($λ$), that integrates novel Fuzzified Eligibility Traces (FET) and Segmented Experience Replay (SER) into fuzzy Q-learning with Fuzzified Bellman Equation (FBE) for continuous control tasks. The proposed approach employs an interpretable fuzzy rule base instead of complex neural architectures, while maintaining competitive performance through two key innovations: a fuzzified Bellman equation with eligibility traces for stable multi-step credit assignment, and a memory-efficient segment-based experience replay mechanism for enhanced sample efficiency. Theoretical analysis proves the proposed method convergence under standard assumptions. Extensive evaluations in continuous control domains demonstrate that Enhanced-FQL($λ$) achieves superior sample efficiency and reduced variance compared to n-step fuzzy TD and fuzzy SARSA($λ$) baselines, while maintaining substantially lower computational complexity than deep RL alternatives such as DDPG. The framework's inherent interpretability, combined with its computational efficiency and theoretical convergence guarantees, makes it particularly suitable for safety-critical applications where transparency and resource constraints are essential.

Figures (2)

• Figure 1: Learning curves comparing Enhanced-FQL($\lambda$) with baseline methods on Cart&Pole benchmark. The proposed method demonstrates faster convergence and superior performance.
• Figure 2: Sample efficiency comparison

Theorems & Definitions (4)

• Definition 1
• Theorem 1
• proof
• Remark 1