Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow

TL;DR

Qflex introduces a value-guided, flow-based exploration mechanism for high-dimensional continuous control. It constructs a Q-guided velocity field $v_Q(\boldsymbol{a};\boldsymbol{s})=\boldsymbol{M}\nabla_{\boldsymbol{a}}Q^{\pi_{old}}(\boldsymbol{s},\boldsymbol{a})$ and transports actions from a learnable source distribution toward high-value regions via an ODE, forming a policy-improvement flow. Flow matching trains the velocity field to align with target transport and integrates seamlessly into an actor-critic loop, preserving full policy expressiveness without dimension reduction. Empirical results across diverse high-dimensional benchmarks and a 700-actuator full-body musculoskeletal model show that Qflex achieves superior sample efficiency and robust, scalable exploration, enabling agile movements like walking, running, and ballet.

Abstract

Controlling high-dimensional systems in biological and robotic applications is challenging due to expansive state-action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcement learning are largely undirected with sharp degradation as action dimensionality grows. Many existing methods resort to dimensionality reduction, which constrains policy expressiveness and forfeits system flexibility. We introduce Q-guided Flow Exploration (Qflex), a scalable reinforcement learning method that conducts exploration directly in the native high-dimensional action space. During training, Qflex traverses actions from a learnable source distribution along a probability flow induced by the learned value function, aligning exploration with task-relevant gradients rather than isotropic noise. Our proposed method substantially outperforms representative online reinforcement learning baselines across diverse high-dimensional continuous-control benchmarks. Qflex also successfully controls a full-body human musculoskeletal model to perform agile, complex movements, demonstrating superior scalability and sample efficiency in very high-dimensional settings. Our results indicate that value-guided flows offer a principled and practical route to exploration at scale.

Figures (9)

• Figure 1: Exploration behavior across increasing action dimensionality. The gray polyline depicts a planar kinematic chain with $|\mathcal{A}|$ degrees of freedom. The orange background (darker is higher) visualizes the state–action value $Q$. Green contours show the end-effector distribution induced by an undirected Gaussian proposal over joint angles, whose exploratory reach collapses as $|\mathcal{A}|$ increases. Red streamlines/contours depict Q-guided probability flows that transport probability mass from the Gaussian proposal toward high-value modes, sustaining directed exploration in high dimensions.
• Figure 2: Control over high-dimensional control benchmarks. (a) Morphologies and state-action dimensions of evaluated benchmarks. (b) Learning curve of algorithms. Results show mean performances with one standard deviation of 5 independent runs. Baselines in the second row are run only on musculoskeletal benchmarks.
• Figure 3: Control over full-body human musculoskeletal system. (a) Learning efficiency over walking control of MS-Human-700. Results show mean performances with one standard deviation of 5 independent runs. (b) Learned behavior of whole-body running. (c) Learned behavior of ballet dancing.
• Figure 4: Sample quality between Qflex and source Gaussian policy during training. Over-actuated musculoskeletal control tasks are denoted as dash-dotted lines.
• Figure 5: Ablation study over hyperparameters of Qflex. (a) Gradient steps $N$. (b) Step size $\eta$. (c) Euler solving timestep $\Delta t$.

Theorems & Definitions (4)

• Proposition 1
• proof
• Proposition 1
• proof