Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress

Abstract

Most robot manipulation focuses on changing the kinematic state of objects: picking, placing, opening, or rotating them. However, a wide range of real-world manipulation tasks involve a different class of object state change--such as mashing, spreading, or slicing--where the object's physical and visual state evolve progressively without necessarily changing its position. We present SPARTA, the first unified framework for the family of object state change manipulation tasks. Our key insight is that these tasks share a common structural pattern: they involve spatially-progressing, object-centric changes that can be represented as regions transitioning from an actionable to a transformed state. Building on this insight, SPARTA integrates spatially progressing object change segmentation maps, a visual skill to perceive actionable vs. transformed regions for specific object state change tasks, to generate a) structured policy observations that strip away appearance variability, and b) dense rewards that capture incremental progress over time. These are leveraged in two SPARTA policy variants: reinforcement learning for fine-grained control without demonstrations or simulation; and greedy control for fast, lightweight deployment. We validate SPARTA on a real robot for three challenging tasks across 10 diverse real-world objects, achieving significant improvements in training time and accuracy over sparse rewards and visual goal-conditioned baselines. Our results highlight progress-aware visual representations as a versatile foundation for the broader family of object state manipulation tasks. Project website: https://vision.cs.utexas.edu/projects/sparta-robot

Figures (6)

• Figure 1: Top: While most robotic manipulation focuses on rigid-body motion, many real-world tasks involve object state changes such as mashing, spreading, or slicing, where objects are progressively transformed. Bottom: SPARTA leverages spatially-progressing affordance maps of actionable vs. transformed regions, successfully demonstrating how to guide real robot manipulation for this family of tasks.
• Figure 2: Overview of $\textsc{SPARTA}$. At each episode step, our policy takes the current and past SPOC mandikal2025spoc visual-affordance (segmentation) maps as inputs (Sec. \ref{['subsec:spoc']}), along with the robot arm's proprioception data and predicts a displacement action for the arm's end-effector. SPARTA supports two robot policy variants: (a) SPARTA-L (Learning): a reinforcement learning agent trained using a dense reward that measures the progressive change of object regions from actionable (red) to transformed (green) (Sec. \ref{['subsec:sparta-l']}); (b) SPARTA-G (Greedy): selects among 8 discrete directions based on the local density of actionable pixels, producing a fast, greedy policy guided by visual progress (Sec. \ref{['subsec:sparta-g']}).
• Figure 3: Our SPOC affordance map generation pipeline.(a) Grounded-SAM ren2024gsam is used to extract an object mask from the initial frame. (b) Farthest-point sampling generates intra-object regions, classified into actionable or transformed by prompting GPT-4o openai2023gpt4 using color-coded overlays. (c) Once classified, transformed regions are tracked across subsequent frames using DeAOT yang2022deaot to maintain temporal consistency with minimal computation.
• Figure 4: SPARTA significantly outmatches the baselines vis-a-vis substantially transforming actionable regions across diverse objects with varying colors, shapes, and textures.
• Figure 5: Reward curves for bread-spreading task. a) Cumulative episode reward curves: $\textsc{SPARTA}$ produces smooth, incremental rewards aligned with visual progress, while LIV rewards remain unstable throughout the episode, offering poor guidance. b) Training curves: stable, dense feedback drives sample-efficient learning, with $\textsc{SPARTA}$ rapidly improving while Sparse and LIV stagnate.