Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

TL;DR

This work tackles offline reinforcement learning for style-conditioned policies by introducing a unified behavior-style definition that uses subtrajectory labeling via labeling functions. It proposes SCIQL, which combines offline IQL with labeling-based style relabeling and a Gated Advantage Weighted Regression (GAWR) mechanism to optimize task performance while preserving style alignment. The approach addresses distribution shift, sparse style supervision, and the difficulty of explicit projection onto the style-optimal set, achieving superior style alignment and style-conditioned performance across diverse environments. The results demonstrate that integrating value learning with relabeling and a gating mechanism yields robust, high-quality stylized behaviors with favorable joint performance, and the code and datasets are publicly available for reproducibility and benchmarking.

Abstract

We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://sciql-iclr-2026.github.io/.

Figures (19)

• Figure 1: Long term decision making and stitching challenges for style alignment optimization. Achieving movement styles such as high-speed running may require to standing and accelerating, which means navigating through different speed styles and demands long-term decision making. Also, trajectories in $\mathcal{D}$ may not cover all the speed styles, calling for trajectory stitching such as (slow $\rightarrow$ medium) and (medium $\rightarrow$ fast).
• Figure 2: Pareto fronts and hypervolumes of SORL and SCIQL. We compare SORL (in blue) and SCIQL (in red). The shaded areas ( , ) represent the hypervolumes covered by the methods. Markers indicate different trade-off configurations: SORL is evaluated at $\beta=0$ ( ), $\beta=1$ ( ), and $\beta=3$ ( ). SCIQL is evaluated with style only $\lambda$ ( ), style-prioritized $\lambda > r$ ( ), and task-prioritized $r > \lambda$ ( ). SCIQL consistently achieves a larger hypervolume and dominates the Pareto frontier.
• Figure 3: Circle2d environment visualizations.
• Figure 4: Circle2d datasets trajectory visualizations at different percentages. The top row corresponds to the circle2d-inplace-v0 while the bottom row corresponds to the circle2d-navigate-v0
• Figure 5: Circle2d position label visualizations at different percentages.