Strategic Jenga Play via Graph Based Dynamics Modeling

TL;DR

This work addresses the challenge of multi-object manipulation under interdependent dynamics by using Jenga as a testbed and introducing graph-based modeling for two tasks: block selection and block extraction. Block selection is cast as a graph-level binary classifier using a GCN on tower graphs to predict whether removing a candidate block causes collapse, while block extraction leverages a Graph Network-based Simulator to learn tower dynamics and an MPPI controller to safely pull blocks via $\bm{a_t}=(a_x,a_y,a_z)$. The approach yields a 0.74 accuracy on block selection and a 65% extraction success rate in simulation, with improvements over naive baselines and clear directions toward sim-to-real transfer, including fusion of vision and tactile signals. Overall, the paper demonstrates that graph-based representations can capture inter-block dependencies and enable high-frequency, strategic manipulation in complex, contact-rich multi-object settings.

Abstract

Controlled manipulation of multiple objects whose dynamics are closely linked is a challenging problem within contact-rich manipulation, requiring an understanding of how the movement of one will impact the others. Using the Jenga game as a testbed to explore this problem, we graph-based modeling to tackle two different aspects of the task: 1) block selection and 2) block extraction. For block selection, we construct graphs of the Jenga tower and attempt to classify, based on the tower's structure, whether removing a given block will cause the tower to collapse. For block extraction, we train a dynamics model that predicts how all the blocks in the tower will move at each timestep in an extraction trajectory, which we then use in a sampling-based model predictive control loop to safely pull blocks out of the tower with a general-purpose parallel-jaw gripper. We train and evaluate our methods in simulation, demonstrating promising results towards block selection and block extraction on a challenging set of full-sized Jenga towers, even at advanced stages of the game.

Figures (7)

• Figure 1: Successful extraction of a difficult-to-remove block from a Jenga tower on an advanced round (round nine) of the game. The coloration of the blocks in the tower is for visualization purposes only.
• Figure 2: Overview of our graph-based approach to strategic Jenga play. For block selection, we train a classifier that predicts whether removing a given block from a tower, represented as a graph, will cause it to enter a failure state. For block selection, the tower state graph is fed into a dynamics model that predicts the displacement of the tower given an action that attempts to remove a candidate block. The block extraction dynamics model is used as part of an MPPI controller to carefully extract blocks.
• Figure 3: Key frames from a simulation rollout: 1) The tower is rotated so the target block is parallel to the $x$-axis and the robot picks up the push-stick, 2) the robot uses pushes the target block out of the tower slightly, 3) the tower rotates 180° and the robot drops the push stick, 4) the robot grasps the target block, 5) the robot begins pulling the block out in the $-x$-direction, and 6) the robot successfully extracts the block.
• Figure 4: Example recovery and failure cases with bad initial pushes.
• Figure 5: Sample tower initializations at different rounds of the game. The coloration of the blocks in the tower is for visualization purposes only.