RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models

TL;DR

This paper tackles the challenge of automated robot design synthesis by leveraging vision-language models to generate kinematically valid, visually realistic articulated robots from simple textual prompts and reference images. The proposed RobotDesignGPT pipeline combines structure synthesis, incremental build, automated visual feedback, and optional human feedback to refine designs, reducing manual input to a few iterations. Experiments demonstrate diverse morphologies, plausible motions via trajectory optimization, and a user study indicating meaningful perceptual realism, while ablations highlight the importance of visual and human feedback. The work advances flexible, grammar-free robot design and integrates with standard motion planners, enabling rapid exploration of novel morphologies with practical implications for design workflows and education.

Abstract

Robot design is a nontrivial process that involves careful consideration of multiple criteria, including user specifications, kinematic structures, and visual appearance. Therefore, the design process often relies heavily on domain expertise and significant human effort. The majority of current methods are rule-based, requiring the specification of a grammar or a set of primitive components and modules that can be composed to create a design. We propose a novel automated robot design framework, RobotDesignGPT, that leverages the general knowledge and reasoning capabilities of large pre-trained vision-language models to automate the robot design synthesis process. Our framework synthesizes an initial robot design from a simple user prompt and a reference image. Our novel visual feedback approach allows us to greatly improve the design quality and reduce unnecessary manual feedback. We demonstrate that our framework can design visually appealing and kinematically valid robots inspired by nature, ranging from legged animals to flying creatures. We justify the proposed framework by conducting an ablation study and a user study.

Figures (8)

• Figure 1: Flowchart depicting an overview of our method.
• Figure 2: Articulated robot models created by the proposed method, which include all of the land, sea, and air creatures.
• Figure 3: Kinematic tree of a rabbit robot generated by the first step of our method, structure synthesis.
• Figure 4: Realistic joint actuation mechanisms designed by our method. The top row depicts the tail actuation mechanism of a seahorse robot while the bottom row demonstrates the wing flapping mechanism of a bee robot.
• Figure 5: Ablations. Each column shows one animal (Woodpecker, Hippopotamus, Elephant), and each row shows a condition: initial design, visual feedback, human feedback, and reference (from top to bottom).