Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots

TL;DR

The paper tackles the poor sample efficiency of multi-objective reinforcement learning by introducing Q-Pensieve, a memory-sharing mechanism that stores past Q-snapshots to inform current policy updates across multiple preferences. By coupling a Q-snapshot replay buffer with a soft-policy iteration framework and an off-policy MOSL actor-critic (MOSAC-like) implementation, the approach preserves convergence guarantees while enabling data reuse at the policy level. The key contributions include the Q-Pensieve policy iteration, the Q replay buffer, and a practical deep MORL algorithm that achieves superior sample efficiency on benchmarks like Deep Sea Treasure, LunarLander, and MuJoCo, along with comprehensive ablations. Overall, the method demonstrates that explicit policy-level memory sharing can significantly reduce the data needs to approximate the Pareto front under linear scalarization, with meaningful improvements for real-world MORL applications.

Abstract

Many real-world continuous control problems are in the dilemma of weighing the pros and cons, multi-objective reinforcement learning (MORL) serves as a generic framework of learning control policies for different preferences over objectives. However, the existing MORL methods either rely on multiple passes of explicit search for finding the Pareto front and therefore are not sample-efficient, or utilizes a shared policy network for coarse knowledge sharing among policies. To boost the sample efficiency of MORL, we propose Q-Pensieve, a policy improvement scheme that stores a collection of Q-snapshots to jointly determine the policy update direction and thereby enables data sharing at the policy level. We show that Q-Pensieve can be naturally integrated with soft policy iteration with convergence guarantee. To substantiate this concept, we propose the technique of Q replay buffer, which stores the learned Q-networks from the past iterations, and arrive at a practical actor-critic implementation. Through extensive experiments and an ablation study, we demonstrate that with much fewer samples, the proposed algorithm can outperform the benchmark MORL methods on a variety of MORL benchmark tasks.

Figures (9)

• Figure 1: The architecture of $\mathop{\mathrm{\mathbf{Q}}}\nolimits$-Pensieve.
• Figure 2: Return vectors attained by $\mathop{\mathrm{\mathbf{Q}}}\nolimits$-Pensieve and the collection of single-objective SAC models under 19 preferences.
• Figure 3: Comparison of standard single-objective SAC and the hybrid SAC assisted by another $Q$-Pensieve model trained in parallel.
• Figure 4: Return vectors attained under preference $\blambda=[0.5,0.5]$ at different training stages (We also plot return vectors under others preference in Figure \ref{['fig:Pareto_growth91']} and Figure \ref{['fig:Pareto_growth19']} in Appendix). A number $x$ on the red or blue marker indicates that the model is obtained at $100\cdot x$ thousand steps.
• Figure 5: A comparison in HV between $\mathop{\mathrm{\mathbf{Q}}}\nolimits$-Pensieve with buffer size equal to 4 and that without using $\mathop{\mathrm{\mathbf{Q}}}\nolimits$ replay buffer at different training stages.

Theorems & Definitions (4)

• Remark 1
• Theorem 1
• Proof : Lemma \ref{['lemma:Soft policy evaluation']}
• Proof : Lemma \ref{['lemma:Soft Policy Improvement']}