Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets

TL;DR

This work tackles the challenge of training goal-conditioned GFlowNets under extremely sparse rewards by introducing Retrospective Backward Synthesis (RBS), which synthesizes backward trajectories from target goals to augment training data with diverse, high-quality experiences. RBS combines forward trajectories with backward-imagined ones, uses age-based replay and backward policy regularization, and intensifies reward signals to stabilize and accelerate learning, with a final loss that couples GC-GFlowNets objectives to a KL-regularized backward policy. Empirically, RBS-GFN achieves state-of-the-art sample efficiency and generalization on GridWorld and sequence-generation tasks, including offline scenarios and challenging unseen-goal/environments, and demonstrates robustness across different GC-GFN objectives (e.g., SubTB) and downstream finetuning. The approach significantly narrows the gap between goal-directed sampling and scalable, data-efficient training in high-dimensional, discrete domains, with potential extensions to hierarchical task decomposition and continuous settings.

Abstract

Generative Flow Networks (GFlowNets), a new family of probabilistic samplers, have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast to standard return maximization approaches (e.g., reinforcement learning) which often converge to a single optimal solution. Recent works have focused on developing goal-conditioned GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly in high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data is available. In this work, we propose a novel method called \textbf{R}etrospective \textbf{B}ackward \textbf{S}ynthesis (\textbf{RBS}) to address these critical problems. Specifically, RBS synthesizes new backward trajectories in goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks.

Figures (18)

• Figure 1: Succee rates with increasing set sizes in set generation.
• Figure 2: Overview of the Retrospective Backward Synthesis (RBS) approach.
• Figure 3: Performance comparison in GridWorld. Left: Small. Middle: Medium. Right: Large.
• Figure 4: Results of RBS-GFN (SubTB).
• Figure 5: Success rates on GridWorld tasks with different sizes.