GFlowNet Foundations

TL;DR

GFlowNets reform the sampling problem by learning flows on a constructively built DAG, ensuring the terminating-state distribution is proportional to a target reward $R(s)$. The framework unifies Markovian-flow representations, detailed-balance style training, and flow-matching losses, enabling amortized sampling that can outperform traditional MCMC in exploring multimodal spaces. The paper extends GFlowNets to conditional and marginalization tasks, including entropy estimation, sampling over sets and graphs, energy-based model training, and Pareto-front exploration, while addressing continuous and stochastic extensions. These theoretical foundations pave the way for versatile applications in active learning, Bayesian inference, and probabilistic reasoning, with open questions around continuous-action regimes, offline/online training, and modular energy-function integration.

Abstract

Generative Flow Networks (GFlowNets) have been introduced as a method to sample a diverse set of candidates in an active learning context, with a training objective that makes them approximately sample in proportion to a given reward function. In this paper, we show a number of additional theoretical properties of GFlowNets. They can be used to estimate joint probability distributions and the corresponding marginal distributions where some variables are unspecified and, of particular interest, can represent distributions over composite objects like sets and graphs. GFlowNets amortize the work typically done by computationally expensive MCMC methods in a single but trained generative pass. They could also be used to estimate partition functions and free energies, conditional probabilities of supersets (supergraphs) given a subset (subgraph), as well as marginal distributions over all supersets (supergraphs) of a given set (graph). We introduce variations enabling the estimation of entropy and mutual information, sampling from a Pareto frontier, connections to reward-maximizing policies, and extensions to stochastic environments, continuous actions and modular energy functions.

Figures (6)

• Figure 1: A diagram of how a GFlowNet iteratively constructs an object. We adopt notation that is common in the reinforcement learning literature: $s_t$ represents the state of the partially constructed object (in this case, a graph) at time $t$, $a_t$ represents the action taken by the GFlowNet at time $t$ to transition to state $s_{t+1}= T(s_t, a_t)$. In this diagram, the GFlowNet takes a 3-node graph as input and determines an action to take. The action, combined with the environment transition function $T(s_t, a_t)$, determines $s_{t+1}$: a four-node graph. This process repeats until an exit action is sampled and the sample is complete.
• Figure 2: Illustration of the structure of a Generative Flow Network (GFlowNet), as a pointed DAG over states $s$, with particles flowing along edges to represent the flow function. Any object sampled by the GFlowNet policy can be obtained by starting from initial state $s_0$ and then at each step choosing a child with probability proportional to the GFlowNet policy's transition probability. This process stops when a terminating action is chosen from a terminating state $s$ (yielding a terminal state $s_f$), at which point a reward $R(s)$ is obtained. The figure shows a tiny GFlowNet and the possible trajectories from $s_0$ to any of the terminal states. It illustrates that in general a state can be reached through several trajectories. GFlowNet algorithms learn a policy such that the probability of sampling terminating state $s$ is proportional to $R(s)$. It tries to learn a flow function $F(s)$ and $F(s\rightarrow s')$ over all states (including intermediate states) $s$ and transitions $s\rightarrow s'$ with $F(s)=R(s)$ at terminal states and $F(s_0)$ being the sum of rewards over all terminal states. A sufficient property to achieve this is that at each state the sum of incoming flows equals the sum of outgoing flows.
• Figure 3: Example of a pointed DAG $G$ illustrating the notions of initial state ($s_0$), final or sink state ($s_f$), terminating states in ${\mathcal{S}}^{f}$, with a transition to $s_f$ called a terminating edge, in ${\mathbb{A}}^{f}$. A terminating state may have other children different from the sink state (e.g., the terminating state $s_{7}$).
• Figure 4: Equivalent flows and Markovian flows. Flows $F_1$ and $F_2$ are equivalent. $F_3$ and $F_4$ are equivalent, but not equivalent to $F_1$ and $F_2$. $F_2$ and $F_4$ are Markovian. $F_1$ and $F_3$ are not Markovian. $F_1, F_2, F_3$ and $F_4$ coincide on the terminating flows i.e. at $s_2 \rightarrow s_f$ and $s_3 \rightarrow s_f$.
• Figure 5: Example of a state-conditional flow network. (a) The original (Markovian) flow network. (b) The subgraph of states reachable from $s_{2}$; there is a flow through $(s_{0}, s_{1}, s_{5})$ that contributed to $F(s_{5}{\rightarrow}s_{f})$, but not to $F(s_{2})$, showing that $F(s_{2})$ does not marginalize the rewards of its descendant. (c) State-conditional flow network $F_{s_{2}}$, which differs from the original flow $F$ on the subgraph, but satisfies the desired marginalization property.

Theorems & Definitions (71)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Lemma 5
• Definition 6
• Definition 7
• Proposition 8
• Definition 9
• Proposition 10