Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes

TL;DR

Talk2Move addresses the challenge of text-guided, object-level geometric transformations in scenes by casting diffusion-based editing as a flow-guided reinforcement learning problem using Group Relative Policy Optimization. A spatially grounded reward disentangles the target object from the background, enabling precise translations, rotations, and rescalings without extensive paired data. The method introduces an efficient training pipeline with off-policy step evaluation and step-wise active sampling to dramatically reduce compute while maintaining performance. Empirical results on synthetic and real images show superior spatial accuracy and scene coherence compared to state-of-the-art editing methods, demonstrating the practical viability of RL-guided geometric scene editing. This approach offers a data-efficient, controllable paradigm for language-driven visual editing with interpretable geometry-aware objectives.

Abstract

We introduce Talk2Move, a reinforcement learning (RL) based diffusion framework for text-instructed spatial transformation of objects within scenes. Spatially manipulating objects in a scene through natural language poses a challenge for multimodal generation systems. While existing text-based manipulation methods can adjust appearance or style, they struggle to perform object-level geometric transformations-such as translating, rotating, or resizing objects-due to scarce paired supervision and pixel-level optimization limits. Talk2Move employs Group Relative Policy Optimization (GRPO) to explore geometric actions through diverse rollouts generated from input images and lightweight textual variations, removing the need for costly paired data. A spatial reward guided model aligns geometric transformations with linguistic description, while off-policy step evaluation and active step sampling improve learning efficiency by focusing on informative transformation stages. Furthermore, we design object-centric spatial rewards that evaluate displacement, rotation, and scaling behaviors directly, enabling interpretable and coherent transformations. Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations, outperforming existing text-guided editing approaches in both spatial accuracy and scene coherence.

Figures (5)

• Figure 1: We introduce Talk2Move, a text-guided scene editing model for object-level geometric transformation, focusing on object translation, rotation and resizing, achieving superior results over current SOTA image editing models.
• Figure 2: The pipeline of Talk2Move.Talk2Move streamlines a GRPO-style reinforcement learning pipeline tailored for flow-based image editing. Starting from an initial noise sample, stochastic perturbations are injected at each diffusion step to generate diverse sampling trajectories. Spatially grounded rewards from specialist models, which explicitly evaluate object-level geometric changes, are then used to compute group-relative advantages for policy gradient updates.
• Figure 3: Three types of step sampling: (a) is the full sampling and optimizing GRPO flowgrpodancegrpo; subsequent methods mixgrpobranchgrpo as in (b), use a sliding window (yellow) to reduce the optimizing steps per iteration; (c) our work introduces step-wise active sampling that select the informative steps (red) and use shortcuts to bypass the rest of the steps, reducing both the sampling and optimizing time.
• Figure 4: Reward behavior across tasks: (a) reward variance distribution, (b) GRPO sampling strategies, and (c) reward model ablations.
• Figure 5: Qualitative results on object translation, rotation and resize over state-of-the-art image editing models. For each task, we provide one real image editing result (source from OpenImagesV6 OpenImages2) and one synthetic image editing result to showcase the generalization ability of Talk2Move.