Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signal from Structure: Exploiting Submodular Upper Bounds in Generative Flow Networks

Alexandre Larouche, Audrey Durand

TL;DR

This work tackles the challenge of sampling from large combinatorial spaces when the reward is unknown but structured. It shows that submodularity induces upper bounds on rewards for unobserved terminating states, enabling data augmentation via a graph-based bound-generation mechanism, and leverages Optimism in the Face of Uncertainty to train GFNs with these bounds. The proposed SuBo-GFN framework achieves orders-of-magnitude more learning signals per reward query than classical GFNs, improving distribution matching while maintaining high-quality candidate generation. Empirical results on synthetic and real-world submodular tasks demonstrate robust data-efficiency, favorable exploration, and scalable performance, suggesting broad applicability to problems like sensor selection and facility location where submodular rewards arise.

Abstract

Generative Flow Networks (GFlowNets; GFNs) are a class of generative models that learn to sample compositional objects proportionally to their a priori unknown value, their reward. We focus on the case where the reward has a specified, actionable structure, namely that it is submodular. We show submodularity can be harnessed to retrieve upper bounds on the reward of compositional objects that have not yet been observed. We provide in-depth analyses of the probability of such bounds occurring, as well as how many unobserved compositional objects can be covered by a bound. Following the Optimism in the Face of Uncertainty principle, we then introduce SUBo-GFN, which uses the submodular upper bounds to train a GFN. We show that SUBo-GFN generates orders of magnitude more training data than classical GFNs for the same number of queries to the reward function. We demonstrate the effectiveness of SUBo-GFN in terms of distribution matching and high-quality candidate generation on synthetic and real-world submodular tasks.

Signal from Structure: Exploiting Submodular Upper Bounds in Generative Flow Networks

TL;DR

This work tackles the challenge of sampling from large combinatorial spaces when the reward is unknown but structured. It shows that submodularity induces upper bounds on rewards for unobserved terminating states, enabling data augmentation via a graph-based bound-generation mechanism, and leverages Optimism in the Face of Uncertainty to train GFNs with these bounds. The proposed SuBo-GFN framework achieves orders-of-magnitude more learning signals per reward query than classical GFNs, improving distribution matching while maintaining high-quality candidate generation. Empirical results on synthetic and real-world submodular tasks demonstrate robust data-efficiency, favorable exploration, and scalable performance, suggesting broad applicability to problems like sensor selection and facility location where submodular rewards arise.

Abstract

Generative Flow Networks (GFlowNets; GFNs) are a class of generative models that learn to sample compositional objects proportionally to their a priori unknown value, their reward. We focus on the case where the reward has a specified, actionable structure, namely that it is submodular. We show submodularity can be harnessed to retrieve upper bounds on the reward of compositional objects that have not yet been observed. We provide in-depth analyses of the probability of such bounds occurring, as well as how many unobserved compositional objects can be covered by a bound. Following the Optimism in the Face of Uncertainty principle, we then introduce SUBo-GFN, which uses the submodular upper bounds to train a GFN. We show that SUBo-GFN generates orders of magnitude more training data than classical GFNs for the same number of queries to the reward function. We demonstrate the effectiveness of SUBo-GFN in terms of distribution matching and high-quality candidate generation on synthetic and real-world submodular tasks.
Paper Structure (55 sections, 23 theorems, 55 equations, 14 figures, 1 table)

This paper contains 55 sections, 23 theorems, 55 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Works
  4. Optimism in the Face of Uncertainty
  5. Improving GFN Exploration
  6. GFNs with Intermediate Learning Signal
  7. Problem Setting
  8. Generative Flow Networks
  9. Submodular Upper Bounds
  10. Generating Upper Bounds
  11. Trajectory-paring graph
  12. Expected Coverage
  13. SuBo-GFN
  14. Data Efficiency
  15. Sampling Distribution Bias
  16. ...and 40 more sections

Key Result

Proposition 4.5

Let $Q(m)$ be the number of distinct bounds on any given terminal state $x$ given a dataset of $m$ trajectories (Assumption ass:uniform_pf), then Note that this is true for any terminating state $x$. Hence, every terminating state have the same expected number of distinct bounds, which grows with $m$, the number of uniformly collected trajectories.

Figures (14)

  • Figure 1: An example of a DAG with submodular reward. States are sets of elements, actions add an element to the current state. Submodularity constrains the reward of states, ensuring that there is a diminishing return over time for the reward of later states.
  • Figure 2: A subset of the DAG, showing an example pair of trajectories for $\textcolor{red}{\mathcal{T}(p_{x,c})}$ and $\textcolor{blue}{\tilde{\mathcal{T}}(p_{x,c})}$ for $x = \{a,b,c\}$. This pair of trajectory yields the bound $\text{UB}(\{a,b,c\}| \emptyset, c) = R(\{c\}) - R(\emptyset) + R(\{a,b\})$. The $\dots$ indicate an ellipsis in the DAG, where any number of valid transitions are taken. Bold edges and vertices highlight key elements in trajectory.
  • Figure 3: Normalized coverage (using Theorem \ref{['thm:true_Q_proba']}) as a function of the cardinality-actions ratio and the number of uniformely sampled trajectories (Assumption \ref{['ass:uniform_pf']}). As $C/N$ decreases, submodular upper bounds covers order of magnitudes more terminating states than would a classical GFN. The black dashed line represents the $1:1$ ratio between query and coverage. Points above this line represent scenarios where SuBo-GFN is worthwhile.
  • Figure 4: Performance metrics and training loss of the classical GFN and SuBo-GFN variants on random graphs. Strategies are trained online (plain line) until the marker, then trained offline (dashed line) without further queries to $R$ until the end of the experiment indicated by the $\hbox{o}rigin=c]{45}{$∎$}$ marker.
  • Figure 5: Performance metrics of the classical GFN and SuBo-GFN variants on real-world graphs. Strategies are trained online (plain line) until the , then trained offline (dashed line) without further queries to $R$ until the end of the experiment indicated by the $\hbox{o}rigin=c]{45}{$∎$}$.
  • ...and 9 more figures

Theorems & Definitions (47)

  • Definition 4.3: Parent trajectories
  • Definition 4.4: Compatible trajectories
  • Proposition 4.5: Expected number of distinct upper bounds on the reward of a terminating state
  • Theorem 4.6: Probability of producing a submodular upper bound
  • Corollary 4.7: Expected coverage of terminating state space
  • Theorem 5.1: Optimism-Induced Oversampling
  • Lemma 1.2
  • proof
  • Proposition 1.3: Number of trajectories passing through a parent
  • proof
  • ...and 37 more