Order-Preserving GFlowNets
Yihang Chen, Lukas Mauch
TL;DR
OP-GFNs address the core limitation of traditional GFlowNets by learning a reward that preserves a (partial) order over candidates rather than relying on a predefined scalar. The framework splits training into an order-preserving loss and MDP constraint losses, yielding a reward landscape that becomes sparser around top candidates as training progresses, which balances exploration early and exploitation later. Theoretical results show the learned reward concentrates on higher-ranked substructures, and extensive experiments across HyperGrid, molecular design, and NAS demonstrate state-of-the-art performance in both single-objective maximization and multi-objective Pareto front approximation, without requiring scalarization. This approach offers a practical, scalable path for diverse candidate generation in expensive or partially observable objective settings, with broad implications for automated design tasks.
Abstract
Generative Flow Networks (GFlowNets) have been introduced as a method to sample a diverse set of candidates with probabilities proportional to a given reward. However, GFlowNets can only be used with a predefined scalar reward, which can be either computationally expensive or not directly accessible, in the case of multi-objective optimization (MOO) tasks for example. Moreover, to prioritize identifying high-reward candidates, the conventional practice is to raise the reward to a higher exponent, the optimal choice of which may vary across different environments. To address these issues, we propose Order-Preserving GFlowNets (OP-GFNs), which sample with probabilities in proportion to a learned reward function that is consistent with a provided (partial) order on the candidates, thus eliminating the need for an explicit formulation of the reward function. We theoretically prove that the training process of OP-GFNs gradually sparsifies the learned reward landscape in single-objective maximization tasks. The sparsification concentrates on candidates of a higher hierarchy in the ordering, ensuring exploration at the beginning and exploitation towards the end of the training. We demonstrate OP-GFN's state-of-the-art performance in single-objective maximization (totally ordered) and multi-objective Pareto front approximation (partially ordered) tasks, including synthetic datasets, molecule generation, and neural architecture search.