RMLer: Synthesizing Novel Objects across Diverse Categories via Reinforcement Mixing Learning

TL;DR

RMLer reframes cross-category concept fusion in text-to-image synthesis as a reinforcement learning problem, using mixed text embeddings as states, interpolation coefficients as actions, and a CLIP-based visual reward to optimize a policy via PPO. A two-stage sampling strategy and foreground-aware rewards ensure semantically balanced, coherent, and visually high-quality fused outputs. The approach achieves state-of-the-art fusion quality on ImageNet-200 and CangJie-200 benchmarks, with strong human preferences and robust ablations demonstrating the necessity of adaptive policy learning. The work provides a robust, modular framework for generating novel visual concepts with potential applications in film, gaming, and design, while outlining limitations and directions for further improvement.

Abstract

Novel object synthesis by integrating distinct textual concepts from diverse categories remains a significant challenge in Text-to-Image (T2I) generation. Existing methods often suffer from insufficient concept mixing, lack of rigorous evaluation, and suboptimal outputs-manifesting as conceptual imbalance, superficial combinations, or mere juxtapositions. To address these limitations, we propose Reinforcement Mixing Learning (RMLer), a framework that formulates cross-category concept fusion as a reinforcement learning problem: mixed features serve as states, mixing strategies as actions, and visual outcomes as rewards. Specifically, we design an MLP-policy network to predict dynamic coefficients for blending cross-category text embeddings. We further introduce visual rewards based on (1) semantic similarity and (2) compositional balance between the fused object and its constituent concepts, optimizing the policy via proximal policy optimization. At inference, a selection strategy leverages these rewards to curate the highest-quality fused objects. Extensive experiments demonstrate RMLer's superiority in synthesizing coherent, high-fidelity objects from diverse categories, outperforming existing methods. Our work provides a robust framework for generating novel visual concepts, with promising applications in film, gaming, and design.

Figures (23)

• Figure 1: We propose a simple yet effective reinforcement mixing learning approach for generating novel object images by fusing distinct categories. For instance, our method seamlessly combines the Venom character with diverse animal categories—such as bulldog, crocodile, turtle, kangaroo, and frog—effectively blending their features to demonstrate its versatility.
• Figure 2: Failures in concept fusion by existing methods. Left (SDXL-Turbo podell2023sdxl): Severe imbalance (e.g., frog + hog$\rightarrow$ dominant frog). Middle (GPT-Image-1): Superficial combination (e.g., pineapple + kangaroo). Right (BASS li2024tp2o): Simple juxtaposition (e.g., owl + snail). Our approach (rightmost) aims for more balanced and coherent fusions.
• Figure 3: Pipeline of our Reinforcement Mixing Learning (RMLer). Given CLIP embeddings for two concepts ($\mathbf{e}_1, \mathbf{e}_2$) extracted from labels ($c_1, c_2$), our policy network $\pi_{\theta}$ generates an action vector $\mathbf{a}$ that mixs $\mathbf{e}_1$ and $\mathbf{e}_2$ into a fused embedding $\mathbf{e}_f$. This embedding conditions a diffusion model $\mathcal{G}$ to synthesize the image $I_f$. A visual reward $R$, computed from CLIP similarity and balance between $I_f$ and references $I_1$ and $I_2$ generated by $\mathbf{e}_1$ and $\mathbf{e}_2$ respectively, guides the PPO algorithm to update $\pi_{\theta}$.
• Figure 4: Comparisons with different methods on ImageNet-200. The complex prompts are created from RMLer-generated image using GPT-4o. For instance, A hybrid creature combining an owl and a snail, with an owl-like head, sharp eyes and a curved beak, and a body covered by a spiral shell texture, standing on a wooden branch with bird-like legs and claws.
• Figure 5: Comparison with different methods on the CangJie-200. The complex prompts are created from RMLer-generated image using GPT-4o. For instance, A hybrid creature combining a butterfly and a chicken, with a compact, feathery bird body, thin legs and claws, and large, vibrant butterfly wings extending from its back, standing on dry forest ground.