CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph

TL;DR

The paper tackles mobile robot exploration in large, occluded environments by introducing a 3D relational object graph (defined as $G=(V,E)$) that encodes semantic and geometric object information and five occlusion relations. CuriousBot combines SLAM, a Graph Constructor, an LLM-driven Task Planner, and a library of Low-Level Skills to enable exploration through active interaction and manipulation, rather than passive perception alone. By leveraging Visual Foundational Models for 3D graph construction and an LLM for planning, the approach achieves robust generalization across diverse scenes and object types, outperforming vision-language baselines. The work demonstrates practical benefits for autonomous exploration and manipulation, while acknowledging the need for scalable skill acquisition and richer semantic representations for future improvements.

Abstract

Mobile exploration is a longstanding challenge in robotics, yet current methods primarily focus on active perception instead of active interaction, limiting the robot's ability to interact with and fully explore its environment. Existing robotic exploration approaches via active interaction are often restricted to tabletop scenes, neglecting the unique challenges posed by mobile exploration, such as large exploration spaces, complex action spaces, and diverse object relations. In this work, we introduce a 3D relational object graph that encodes diverse object relations and enables exploration through active interaction. We develop a system based on this representation and evaluate it across diverse scenes. Our qualitative and quantitative results demonstrate the system's effectiveness and generalization capabilities, outperforming methods that rely solely on vision-language models (VLMs).

Figures (6)

• Figure 1: CuriousBot. We present a mobile robotic system that can (a) interactively explore the environment, such as inspecting hidden spaces inside a cabinet or behind a box, (b) construct an actionable 3D relational object graph that encodes both the semantic and geometric information of object nodes, along with various object relationships, and (c) perform manipulation tasks by retrieving objects through traversal of the actionable 3D relational object graph.
• Figure 2: Method Overview. (a) In the perception pipeline, SLAM processes RGBD observations and odometry estimation from the robot to output camera poses, which are used alongside the RGBD observations to construct an actionable 3D relational object graph. (b) The 3D relational object graph comprises object nodes containing both geometric and semantic information, as well as object edges that encode complex object relations. (c) The serialized object graph is fed into the task planner, and the generated task plans are executed using low-level skills to interactively explore the environment.
• Figure 3: Experiment Setup. (a) illustrates the use of a Spot robot equipped with an external RealSense 455. (b) showcases the diverse objects used, emphasizing the system's generalization capabilities across various object types, scenes, and object relations.
• Figure 4: Diverse Scenes and Skills. We evaluate our system's exploration capabilities across various tasks, including pushing the chair aside to reveal space behind it, lifting cloth to check underneath, flipping open boxes to inspect the contents, and exploring a household scene. These tasks showcase the system's ability to generalize across different object types, scenarios, and object relations. Additional tasks can be found on the https://bdaiinstitute.github.io/curiousbot/.
• Figure 5: Failure Breakdown. We analyze the failure modes of our system during exploration tasks, identifying three main causes: perception failure, decision failure, and action failure.