Knolling Bot: Teaching Robots the Human Notion of Tidiness

TL;DR

This work addresses the challenge of endowing robots with a human-like sense of tidiness for home environments. It treats knolling as an autoregressive sequence prediction problem and uses a transformer architecture paired with a Gaussian Mixture Model to capture multiple valid object placements, enabling diverse, preference-aware arrangements. The approach is trained in a self-supervised manner on a large synthetic dataset and integrated into a complete pipeline with a perception module and robotic controller, achieving real-world tidying with varying object counts. The authors also release a dataset and benchmark to foster reproducibility and further study of object rearrangement with arbitrary numbers and shapes, advancing the goal of collaborative, aesthetically aware robotic assistants in living spaces.

Abstract

For robots to truly collaborate and assist humans, they must understand not only logic and instructions, but also the subtle emotions, aesthetics, and feelings that define our humanity. Human art and aesthetics are among the most elusive concepts-often difficult even for people to articulate-and without grasping these fundamentals, robots will be unable to help in many spheres of daily life. Consider the long-promised robotic butler: automating domestic chores demands more than motion planning. It requires an internal model of cleanliness and tidiness-a challenge largely unexplored by AI. To bridge this gap, we propose an approach that equips domestic robots to perform simple tidying tasks via knolling, the practice of arranging scattered items into neat, space-efficient layouts. Unlike the uniformity of industrial settings, household environments feature diverse objects and highly subjective notions of tidiness. Drawing inspiration from NLP, we treat knolling as a sequential prediction problem and employ a transformer based model to forecast each object's placement. Our method learns a generalizable concept of tidiness, generates diverse solutions adaptable to varying object sets, and incorporates human preferences for personalized arrangements. This work represents a step forward in building robots that internalize human aesthetic sense and can genuinely co-create in our living spaces.

Figures (7)

• Figure 1: Examples from the knolling task. A) A batch of small items, including daily necessities. B) A batch of big items fabricated by a 3D printer. C) Diverse knolling preferences demonstrated in experiments. The model adapts its tidying behavior based on different preferences: object category (group by function), size (large objects first), and color (group by hue).
• Figure 2: A) Challenges in rearrangement tasks with multiple solutions. From left to right: the initial state of the work area with an unplaced yellow motor. Three proposed placement options (1, 2, 3) for the yellow motor, considering factors such as object category, color similarity, and spatial efficiency. Placement by a regression-based model optimizing for minimal loss across three solutions, leading to undesirable results: placing the motor on a utility knife. B) Knolling pipeline. The left side of the figure displays a cluttered desktop with various objects such as batteries, erasers, electronic components, and other daily necessities in the lab. Our robot initiates the knolling process after detecting and identifying these objects through a camera. The right side of the figure depicts the outcome of this process, presenting a tidy, well-organized desktop. This transformation exemplifies the robot's ability to apply the knolling model, execute a tidying task, and create a pleasing and space-efficient arrangement.
• Figure 3: Knolling Model Learning Framework: The pipeline begins with the visual perception model, which processes an input image to identify objects and extract their state representations—including width (w), length (l), position and orientation, presented as a list. However, only w and l are used as input for the knolling. The knolling model takes high-dimensional object states (h) derived from positional encoding as input. During training, a masked learning approach is employed, where part of the object data (M) is masked, and the model learns to predict the next object's position (P). The model predicts N target positions after N iterative processes.
• Figure 4: Examples of knolling messy tables with different numbers of objects. The figure shows ten examples of tables before and after the knolling process in the simulation.
• Figure 5: A) Box knolling in the real world. In each test, we show four columns. Column 1: The initial state of the objects on a table, as captured by the overhead camera. Column 2: The same scenario as Column 1, with added key points and contour outlines indicating the detected objects. Column 3: Action snapshot of the robot executing the knolling task. Column 4: The final state of the workspace post-knolling, presenting an tidy table. B) Real-world Knolling Process with Different Object Numbers. This figure exhibits the practical application of our knolling model in four diverse scenarios. Each column corresponds to a different setup with a distinct number of objects (6, 8, 10, and 10). We show the initial messy state captured by the overhead camera and the organized layout after the knolling task is completed by the robot arm. These comparative visuals underline our robot's proficiency in performing real-world knolling tasks across varied object quantities. C) For the same objects on the table, our robot can perform knolling tasks with different solutions based on preferences based on category, color, or shape.