Break Out the Silverware -- Semantic Understanding of Stored Household Items

TL;DR

The paper defines the Stored Household Item Challenge, a benchmark for inferring the likely storage location of non-visible household items in kitchen scenes. It introduces NOAM, a vision-to-language pipeline that converts visual container cues into natural-language prompts for LLMs to reason about hidden storage and outputs a concrete container polygon evaluated by IoU. Two new datasets (a 6,500-item development set with polygon annotations and a 100-item real-world evaluation set) enable scalable training and robust testing against baselines including vision-only and multimodal models; NOAM significantly outperforms these baselines and approaches human performance. The work discusses efficiency, generalization, and embodied extensions, highlighting practical implications for deploying cognitively capable domestic service robots in cluttered, real-world environments.

Abstract

``Bring me a plate.'' For domestic service robots, this simple command reveals a complex challenge: inferring where everyday items are stored, often out of sight in drawers, cabinets, or closets. Despite advances in vision and manipulation, robots still lack the commonsense reasoning needed to complete this task. We introduce the Stored Household Item Challenge, a benchmark task for evaluating service robots' cognitive capabilities: given a household scene and a queried item, predict its most likely storage location. Our benchmark includes two datasets: (1) a real-world evaluation set of 100 item-image pairs with human-annotated ground truth from participants' kitchens, and (2) a development set of 6,500 item-image pairs annotated with storage polygons over public kitchen images. These datasets support realistic modeling of household organization and enable comparative evaluation across agent architectures. To begin tackling this challenge, we introduce NOAM (Non-visible Object Allocation Model), a hybrid agent pipeline that combines structured scene understanding with large language model inference. NOAM converts visual input into natural language descriptions of spatial context and visible containers, then prompts a language model (e.g., GPT-4) to infer the most likely hidden storage location. This integrated vision-language agent exhibits emergent commonsense reasoning and is designed for modular deployment within broader robotic systems. We evaluate NOAM against baselines including random selection, vision-language pipelines (Grounding-DINO + SAM), leading multimodal models (e.g., Gemini, GPT-4o, Kosmos-2, LLaMA, Qwen), and human performance. NOAM significantly improves prediction accuracy and approaches human-level results, highlighting best practices for deploying cognitively capable agents in domestic environments.

Figures (6)

• Figure 1: Kitchen scene shown to Gemini, with its predicted spoon storage location highlighted in the bounding box.
• Figure 2: Kitchen scene shown to GPT, with its predicted spoon storage location highlighted in the bounding box.
• Figure 3: Depiction of the Stored Household Item Challenge (right). We compare the performance of various models on this task (left), including grounded vision-language models, multimodal LLMs, and our model, NOAM.
• Figure 4: A screenshot from the annotation tool used to collect human-labeled data efficiently.
• Figure 5: An overview of our NOAM pipeline: from an input image (1) through container detection (2), feature extraction (3), textual explanation (4), prompt generation (5), through LLM processing (6) to response (7) and value extraction (8) and overall evaluation (9).