Spline-FRIDA: Towards Diverse, Humanlike Robot Painting Styles with a Sample-Efficient, Differentiable Brush Stroke Model

TL;DR

This work targets the limitation of rigid stroke representations in robot painting by introducing Spline-FRIDA, which records human brush trajectories via motion capture and encodes them with a TrajVAE. A differentiable renderer, Traj2Stroke, turns decoded trajectories into stroke masks with a compact 7-parameter model, enabling end-to-end gradient planning to match target images. Empirical results show that TrajVAE-inspired stroke trajectories yield more human-like, artistic paintings than FRIDA's Bézier-based strokes, supported by human evaluations and quantitative stroke-dynamics experiments. The approach is modular and sample-efficient, improving semantic planning and offering a practical path toward diverse robot painting styles with a low sim2real gap.

Abstract

A painting is more than just a picture on a wall; a painting is a process comprised of many intentional brush strokes, the shapes of which are an important component of a painting's overall style and message. Prior work in modeling brush stroke trajectories either does not work with real-world robotics or is not flexible enough to capture the complexity of human-made brush strokes. In this work, we introduce Spline-FRIDA which can model complex human brush stroke trajectories. This is achieved by recording artists drawing using motion capture, modeling the extracted trajectories with an autoencoder, and introducing a novel brush stroke dynamics model to the existing robotic painting platform FRIDA. We conducted a survey and found that our open-source Spline-FRIDA approach successfully captures the stroke styles in human drawings and that Spline-FRIDA's brush strokes are more human-like, improve semantic planning, and are more artistic compared to existing robot painting systems with restrictive Bézier curve strokes.

Figures (10)

• Figure 1: Spline-FRIDA drawings and paintings in different styles. These are two pairs of artworks of made by our system. The left paintings use longer, zig-zagging strokes, while the right ones are composed of small circles and dots. While each pair depicts the same content, the stroke style vastly changes the appearance and vibe of each work.
• Figure 2: Weaknesses in FRIDA's stroke model. Left: FRIDA struggles with rendering thin strokes. Right: for a full drawing, errors in individual stroke predictions accumulate. The optimization process exploits blotchy predictions made by the renderer to create small dots, which appear quite different when actually executed.
• Figure 3: Traj2Stroke. The inputs are a latent vector $z$ and an offset $\Delta$. $z$ is fed through the decoder of a TrajVAE, generating a raw trajectory, which is then rotated and translated according to $\Delta$. We then process the trajectory segments independently, obtaining darkness values for each. Finally, we take the max darkness over all segments.
• Figure 4: Mocap setup. We use a motion capture system to track the position of the canvas and pen over time as an artist draws. Three mocap markers are placed along the corners of the canvas, and four are mounted at the end of the pen. The trajectories of each stroke are extracted, then rotated and translated such that the start of the trajectory is (0, 0) and the end point is on the x-axis.
• Figure 5: Planning a Painting. As described in \ref{['sec:planning']}, Spline-FRIDA plans a painting by optimizing the brush stroke parameters through the dynamics model to decrease a features space loss between a given image and the planned painting. Whereas FRIDA models brush strokes as simple Bézier curves, Spline-FRIDA uses trajectories which enable highly flexible brush strokes.