Sat-EnQ: Satisficing Ensembles of Weak Q-Learners for Reliable and Compute-Efficient Reinforcement Learning
Ünver Çiftçi
TL;DR
Sat-EnQ tackles deep Q-learning instability by introducing a two-phase approach: first train an ensemble of lightweight, satisficing Q-learners with a dynamic baseline to produce bounded, low-variance estimates, then distill these into a larger network for fine-tuning with standard Double DQN. The paper proves that the satisficing operator bounds updates and cannot increase target variance, and demonstrates substantial variance reduction and robustness to environmental noise with notable compute savings compared to bootstrapped ensembles. Empirically, Sat-EnQ achieves up to 3.8× variance reduction, eliminates catastrophic failures, and maintains performance under action noise, while reducing training compute by about 2.5×. This work highlights the value of bounded rationality and coarse-to-fine learning for robust RL and suggests future directions for adaptive margins and sparse-reward scenarios.
Abstract
Deep Q-learning algorithms remain notoriously unstable, especially during early training when the maximization operator amplifies estimation errors. Inspired by bounded rationality theory and developmental learning, we introduce Sat-EnQ, a two-phase framework that first learns to be ``good enough'' before optimizing aggressively. In Phase 1, we train an ensemble of lightweight Q-networks under a satisficing objective that limits early value growth using a dynamic baseline, producing diverse, low-variance estimates while avoiding catastrophic overestimation. In Phase 2, the ensemble is distilled into a larger network and fine-tuned with standard Double DQN. We prove theoretically that satisficing induces bounded updates and cannot increase target variance, with a corollary quantifying conditions for substantial reduction. Empirically, Sat-EnQ achieves 3.8x variance reduction, eliminates catastrophic failures (0% vs 50% for DQN), maintains 79% performance under environmental noise}, and requires 2.5x less compute than bootstrapped ensembles. Our results highlight a principled path toward robust reinforcement learning by embracing satisficing before optimization.