Active Causal Experimentalist (ACE): Learning Intervention Strategies via Direct Preference Optimization

TL;DR

ACE tackles the problem of learning efficient sequential experimental designs for causal discovery by casting intervention selection as a policy trained with Direct Preference Optimization. By focusing on pairwise intervention preferences and a reward that combines information gain, node importance, and diversity, ACE learns strategies that adapt as knowledge accumulates, avoiding the instability of value-based rewards. The approach yields substantial improvements (about 70–71% over baselines at equal budgets, with p<0.001 and Cohen's d ~ 2) and exhibits emergent, theoretically grounded behaviors such as concentrating interventions on collider parents. Beyond synthetic benchmarks, ACE demonstrates transfer to physics and economics domains, including retrospective causal learning from historical data, highlighting the practical impact of preference-based learning for domain adaptation in scientific discovery.

Abstract

Discovering causal relationships requires controlled experiments, but experimentalists face a sequential decision problem: each intervention reveals information that should inform what to try next. Traditional approaches such as random sampling, greedy information maximization, and round-robin coverage treat each decision in isolation, unable to learn adaptive strategies from experience. We propose Active Causal Experimentalist (ACE), which learns experimental design as a sequential policy. Our key insight is that while absolute information gains diminish as knowledge accumulates (making value-based RL unstable), relative comparisons between candidate interventions remain meaningful throughout. ACE exploits this via Direct Preference Optimization, learning from pairwise intervention comparisons rather than non-stationary reward magnitudes. Across synthetic benchmarks, physics simulations, and economic data, ACE achieves 70-71% improvement over baselines at equal intervention budgets (p < 0.001, Cohen's d ~ 2). Notably, the learned policy autonomously discovers that collider mechanisms require concentrated interventions on parent variables, a theoretically-grounded strategy that emerges purely from experience. This suggests preference-based learning can recover principled experimental strategies, complementing theory with learned domain adaptation.

Figures (6)

• Figure 1: ACE framework overview. The experimentalist $\pi_\phi$ proposes interventions, the environment $M^*$ generates data, and the learner $M_\theta$ updates its mechanism estimates. Per-node losses $\{L_i\}$ inform the next intervention. DPO training uses preference pairs constructed from candidate comparisons.
• Figure 2: ACE algorithm for one experimental step. The policy observes the learner's state and per-node losses, generates $K$ candidate interventions, simulates each on a cloned learner to estimate information gain, selects and executes the best candidate, then updates both the learner (with new data) and the policy (via DPO on preference pairs). Dashed arrows indicate feedback loops.
• Figure 3: Synthetic 5-node benchmark with hierarchical structure. $X_1$ and $X_4$ are roots (blue), $X_2$ and $X_5$ are intermediates (gray), and $X_3$ is a collider (red) with edges from both $X_1$ and $X_2$. The disconnected chain $X_4 \to X_5$ tests quadratic mechanisms.
• Figure 4: Structure of the complex 15-node SCM with 4 roots (blue) and 11 colliders (red). Every endogenous node has exactly two parents, making this a collider-dense structure that challenges experimental design strategies. Nested colliders ($C_4$, $C_5$) at layer 3 test reasoning about causal depth.
• Figure 5: Coupled Duffing oscillators. (a) True chain coupling. (b) Synchronization creates spurious correlation (dashed). ACE discovers clamping $X_2$ breaks spurious $X_1$--$X_3$ correlation.