Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KITchen: A Real-World Benchmark and Dataset for 6D Object Pose Estimation in Kitchen Environments

Abdelrahman Younes, Tamim Asfour

TL;DR

KITchen tackles the gap between lab 6D pose benchmarks and real-world kitchen robotics by providing a large-scale, real-world RGBD dataset of 111 kitchen objects captured from a humanoid robot's egocentric FOV in two kitchens. It introduces a semi-automated annotation pipeline that combines synthetic data generation, YOLOv8-based detection, SAM-based masks, and MegaPose-based 6D pose estimation with manual refinement, yielding 2D boxes, masks, and 6D poses. The accompanying benchmark enforces real-time constraints (≥5 fps) and monocular FOV usage, challenging methods to perform robust multi-object pose estimation in non-centered, cluttered kitchen scenes. Collectively, KITchen enables evaluation and development of real-world, robot-friendly 6D pose estimation methods applicable to mobile manipulation tasks in domestic environments.

Abstract

Despite the recent progress on 6D object pose estimation methods for robotic grasping, a substantial performance gap persists between the capabilities of these methods on existing datasets and their efficacy in real-world grasping and mobile manipulation tasks, particularly when robots rely solely on their monocular egocentric field of view (FOV). Existing real-world datasets primarily focus on table-top grasping scenarios, where a robot arm is placed in a fixed position and the objects are centralized within the FOV of fixed external camera(s). Assessing performance on such datasets may not accurately reflect the challenges encountered in everyday grasping and mobile manipulation tasks within kitchen environments such as retrieving objects from higher shelves, sinks, dishwashers, ovens, refrigerators, or microwaves. To address this gap, we present KITchen, a novel benchmark designed specifically for estimating the 6D poses of objects located in diverse positions within kitchen settings. For this purpose, we recorded a comprehensive dataset comprising around 205k real-world RGBD images for 111 kitchen objects captured in two distinct kitchens, utilizing a humanoid robot with its egocentric perspectives. Subsequently, we developed a semi-automated annotation pipeline, to streamline the labeling process of such datasets, resulting in the generation of 2D object labels, 2D object segmentation masks, and 6D object poses with minimal human effort. The benchmark, the dataset, and the annotation pipeline will be publicly available at https://kitchen-dataset.github.io/KITchen.

KITchen: A Real-World Benchmark and Dataset for 6D Object Pose Estimation in Kitchen Environments

TL;DR

KITchen tackles the gap between lab 6D pose benchmarks and real-world kitchen robotics by providing a large-scale, real-world RGBD dataset of 111 kitchen objects captured from a humanoid robot's egocentric FOV in two kitchens. It introduces a semi-automated annotation pipeline that combines synthetic data generation, YOLOv8-based detection, SAM-based masks, and MegaPose-based 6D pose estimation with manual refinement, yielding 2D boxes, masks, and 6D poses. The accompanying benchmark enforces real-time constraints (≥5 fps) and monocular FOV usage, challenging methods to perform robust multi-object pose estimation in non-centered, cluttered kitchen scenes. Collectively, KITchen enables evaluation and development of real-world, robot-friendly 6D pose estimation methods applicable to mobile manipulation tasks in domestic environments.

Abstract

Despite the recent progress on 6D object pose estimation methods for robotic grasping, a substantial performance gap persists between the capabilities of these methods on existing datasets and their efficacy in real-world grasping and mobile manipulation tasks, particularly when robots rely solely on their monocular egocentric field of view (FOV). Existing real-world datasets primarily focus on table-top grasping scenarios, where a robot arm is placed in a fixed position and the objects are centralized within the FOV of fixed external camera(s). Assessing performance on such datasets may not accurately reflect the challenges encountered in everyday grasping and mobile manipulation tasks within kitchen environments such as retrieving objects from higher shelves, sinks, dishwashers, ovens, refrigerators, or microwaves. To address this gap, we present KITchen, a novel benchmark designed specifically for estimating the 6D poses of objects located in diverse positions within kitchen settings. For this purpose, we recorded a comprehensive dataset comprising around 205k real-world RGBD images for 111 kitchen objects captured in two distinct kitchens, utilizing a humanoid robot with its egocentric perspectives. Subsequently, we developed a semi-automated annotation pipeline, to streamline the labeling process of such datasets, resulting in the generation of 2D object labels, 2D object segmentation masks, and 6D object poses with minimal human effort. The benchmark, the dataset, and the annotation pipeline will be publicly available at https://kitchen-dataset.github.io/KITchen.
Paper Structure (17 sections, 7 figures, 1 table)

This paper contains 17 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Objects Datasets
  4. 6D Pose Estimation Methods
  5. The KITchen Dataset
  6. Dataset's Objects
  7. Dataset Recording
  8. Annotation Pipeline
  9. 2D Objects Bounding Boxes Annotation
  10. 2D Objects Segmentation Masks
  11. 6D Object Poses
  12. Comparison to Existing Datasets
  13. The KITchen Benchmark
  14. Problem Statement
  15. Datasets
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Challenging kitchen locations that our dataset covers in contrast with the currently available datasets. The objects are distributed across diverse locations such as fridge, drawer, sink, higher shelves, microwave, dishwasher, oven, etc.
  • Figure 2: The humanoid robot ARMAR-6, leveraged for its adjustable torso height and various camera angles provided by its adjustable roll-yaw neck, to enrich our dataset.
  • Figure 3: The two distinguished kitchens where we recorded our dataset. On the left side is the Main Kitchen while on the right side is the Mobile Kitchen.
  • Figure 4: Diverse robot and camera heights realized through different torso positions of ARMAR-6. The images display heights of 145cm, 177cm, and 185cm from left to right, illustrating the varied perspectives captured in the datasets and the different placements of objects relative to the robot's field of view.
  • Figure 5: Variation in robot neck pitch angle. The images depict angles of 10, 37, and 49 degrees from left to right, showcasing a diverse range of perspectives.
  • ...and 2 more figures