IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction

Abstract

Robotic reproduction of oil paintings using soft brushes and pigments requires force-sensitive control of deformable tools, prediction of brushstroke effects, and multi-step stroke planning, often without human step-by-step demonstrations or faithful simulators. Given only a sequence of target oil painting images, can a robot infer and execute the stroke trajectories, forces, and colors needed to reproduce it? We present IMPASTO, a robotic oil-painting system that integrates learned pixel dynamics models with model-based planning. The dynamics models predict canvas updates from image observations and parameterized stroke actions; a receding-horizon model predictive control optimizer then plans trajectories and forces, while a force-sensitive controller executes strokes on a 7-DoF robot arm. IMPASTO integrates low-level force control, learned dynamics models, and high-level closed-loop planning, learns solely from robot self-play, and approximates human artists' single-stroke datasets and multi-stroke artworks, outperforming baselines in reproduction accuracy. Project website: https://impasto-robopainting.github.io/

Figures (15)

• Figure 2: Action parameterization. Each brushstroke is represented by a quadratic Bézier curve, with a starting location $p_0=(x_0,y_0)$, length $l$, bend $b$, orientation $\alpha$, force $F$ (controls thickness), and color $C\in[0,1]^4$ (RGBA).
• Figure 3: Hardware system setup of IMPASTO.
• Figure 4: Overview of the learning and planning framework. Top: IMPASTO-UNet's neural pixel dynamics model, which combines an image encoder and an action encoder to predict the effect of a stroke. The model is trained using a weighted $\ell_1$ loss. Bottom: To find one or more consecutive strokes between a base image and a target image, an MPC-based planner optimizes stroke parameters with a weighted $\ell_1$ image objective in a receding-horizon, closed loop.
• Figure 5: Training and evaluation $\mathcal{L}_{\ell_1}$ (unweighted) of IMPASTO's dynamics model vs. number of training samples (log–log scale).
• Figure 6: Target brushstrokes from five human artists (two examples per artist) and the strokes reproduced by the robot using different methods. The numbers shown are the weighted $\ell_1$ loss between the target and the painted strokes. Instances with the best performances are highlighted with bold borders. Overall, IMPASTO-UNet approximated human brushstrokes with lower error and higher visual similarity.