OPA-Pack: Object-Property-Aware Robotic Bin Packing

TL;DR

OPA-Pack addresses a key gap in robotic bin packing by incorporating object properties into planning. It combines retrieval-augmented generation and chain-of-thought reasoning for object property recognition with a property-aware deep Q-learning framework (OPA-Net) that jointly optimizes packing compactness, avoidance of incompatible object pairs, and protection of fragile items. The system builds a new 1,032-object OPA dataset, demonstrates substantial gains in avoidance accuracy and reduced fragility impact, and validates effectiveness on both virtual simulators and a real packing platform. This property-centric approach enables safer, more practical packing in real-world settings where object semantics and fragility matter for operational throughput and product safety.

Abstract

Robotic bin packing aids in a wide range of real-world scenarios such as e-commerce and warehouses. Yet, existing works focus mainly on considering the shape of objects to optimize packing compactness and neglect object properties such as fragility, edibility, and chemistry that humans typically consider when packing objects. This paper presents OPA-Pack (Object-Property-Aware Packing framework), the first framework that equips the robot with object property considerations in planning the object packing. Technical-wise, we develop a novel object property recognition scheme with retrieval-augmented generation and chain-of-thought reasoning, and build a dataset with object property annotations for 1,032 everyday objects. Also, we formulate OPA-Net, aiming to jointly separate incompatible object pairs and reduce pressure on fragile objects, while compacting the packing. Further, OPA-Net consists of a property embedding layer to encode the property of candidate objects to be packed, together with a fragility heightmap and an avoidance heightmap to keep track of the packed objects. Then, we design a reward function and adopt a deep Q-learning scheme to train OPA-Net. Experimental results manifest that OPA-Pack greatly improves the accuracy of separating incompatible object pairs (from 52% to 95%) and largely reduces pressure on fragile objects (by 29.4%), while maintaining good packing compactness. Besides, we demonstrate the effectiveness of OPA-Pack on a real packing platform, showcasing its practicality in real-world scenarios.

Figures (9)

• Figure 1: Illustration of the proposed framework. Our OPA-Pack consists of two stages. In the first stage, we recognize the object properties using a Vision-Language Model (VLM). Retrieval Augmented Generation (RAG) based on CLIP radford2021learning and Chain-of-Thought (CoT) reasoning are used to provide additional information to enhance the recognition. We obtain object-centric properties, e.g., fragility, softness, sharpness, and density levels, and also infer the object relation properties to identify the incompatible object pairs that should not be packed closely with each other. In the second stage, we perform object property-aware packing learning via an OPA-Net to reduce pressure on fragile objects and avoid closely packed incompatible object pairs.
• Figure 2: Our pipeline for object property recognition includes (a) object-centric properties and (b) property-based avoidance relations. For a given object, we provide four views, its class name, and size, and then predict nine object-centric properties (in yellow). To enhance the prediction performance, we provide examples retrieved from a small auxiliary dataset using the CLIP feature, performing RAG (in green). Also, we predict the inner and outer materials of each object and provide additional knowledge of the materials (see Table \ref{['tab:density']}), leveraging CoT reasoning to enhance the prediction of fragility and density levels (in blue). Apart from physical properties, we predict also the semantic properties of the object. Besides, we discover the avoidance relations between each pair of objects indicating the object pairs that are not suitable for close packing, considering four types of avoidance relations, i.e., (i) sharp objects v.s. soft objects, (ii) medicine v.s. edible objects, (iii) household chemicals v.s. edible objects, and (iv) ignition items v.s., flammable items. Best viewed in color.
• Figure 3: Some object examples from our OPA dataset divided into three subsets.
• Figure 4: Illustration of the OPA-Net architecture. The network contains four types of encoders, the shape encoder $E_{\mathbf{P}}$, the pose-embedding layer $E_{\mathbf{O}}$, the property-embedding layer $E_{\mathbf{V}}$, and the heightmap encoders ($E_{C}$$E_{G}$ and $E_{D}$). Then, an object Q-predictor is trained to predict the next object to be packed and a placement Q-predictor is trained to learn the placement of an object. Best viewed in color.
• Figure 5: Visualization of the packing results produced by the baseline zhao2023learning and our method from each of the following subsets: Hardware and Stationeries, Daily Necessities, and Kitchen. In each case, we first display the overall packing results in the first column, followed by a visualization of the fragile objects in the second column, and finally an avoidance object pair in the last column. Compared to the baseline, our method tends to position fragile objects on top to prevent compression from heavier items and more effectively separates avoidance object pairs.