Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies

TL;DR

The paper tackles the challenge that reinforcement learning policies often hit a performance ceiling in complex, multi-agent Dec-POMDP settings. It proposes and comprehensively evaluates inference-time strategies—stochastic sampling, simulation-guided tree search, online fine-tuning, and diversity-based COMPASS—across budgets of time and compute, demonstrating substantial gains (up to 126% and average 45%) over zero-shot baselines on 17 tasks. Through an extensive, large-scale study (60k experiments) with tight, realistic budgets, the authors show that inference-time search can scale with compute and time budgets, offering practical guidance via performance contour maps. The findings argue for treating inference-time strategies as core components of deployment, capable of pushing RL performance closer to real-world optima while maintaining tractable execution costs.

Abstract

Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many digital or simulation-based applications allow for an inference phase that utilises a specific time and compute budget to explore multiple attempts before outputting a final solution. In this work, we show that such an inference phase employed at execution time, and the choice of a corresponding inference strategy, are key to breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain up to a 126% and, on average, a 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during execution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. Our experimental data and code are available at https://sites.google.com/view/inference-strategies-rl.

Figures (12)

• Figure 1: Improvement from using inference-time search over zero-shot state-of-the-art. Across 17 complex reinforcement learning tasks, we obtain consistent and significant performance gains using only a 30 second search budget during execution.
• Figure 2: Numerous applications of RL involve two distinct phases: (1) a training phase, typically unconstrained in time and compute, during which a policy is optimized over a representative distribution of problem instances; and (2) an inference phase, where a limited time and compute budget are allocated to solving a new instance. The inference phase is often overlooked, despite its crucial role in complex tasks where partial observability and the combinatorial growth of observation and action spaces make good solutions unattainable through zero-shot execution alone.
• Figure 3: Overview of our evaluation tasks and experimental study.
• Figure 4: Training the SOTA algorithm Sable to convergence is not enough to achieve optimal zero-shot performance. We report the mean normalised performance per task and report 95% bootstrap confidence intervals in \ref{['app:additional-error-bars']}.
• Figure 5: Performance obtained by inference strategies over the benchmark. Each base policy is evaluated with each possible inference strategy. We report the inter-quartile mean over tasks with 95% stratified bootstrap confidence intervals.