World Knowledge from AI Image Generation for Robot Control

TL;DR

This paper addresses how robots can act under under-specified tasks by leveraging implicit world knowledge embedded in modern generative AI images. It proposes conditioning image generation on the current environment layout to produce goal-state visuals that guide robot actions, using edge maps to preserve scene layout and text prompts to add missing objects. Two experiments in CoppeliaSim demonstrate placing a bowl and hanging a painting, where generated images inform placement via bounding boxes and depth cues. The findings show that generated imagery can capture prototypical object relations and arrangements, offering a scalable way to harness vast web-era knowledge for real-time robotic decision-making, with discussion of integration with multi-modal models for broader capability.

Abstract

When interacting with the world robots face a number of difficult questions, having to make decisions when given under-specified tasks where they need to make choices, often without clearly defined right and wrong answers. Humans, on the other hand, can often rely on their knowledge and experience to fill in the gaps. For example, the simple task of organizing newly bought produce into the fridge involves deciding where to put each thing individually, how to arrange them together meaningfully, e.g. putting related things together, all while there is no clear right and wrong way to accomplish this task. We could encode all this information on how to do such things explicitly into the robots' knowledge base, but this can quickly become overwhelming, considering the number of potential tasks and circumstances the robot could encounter. However, images of the real world often implicitly encode answers to such questions and can show which configurations of objects are meaningful or are usually used by humans. An image of a full fridge can give a lot of information about how things are usually arranged in relation to each other and the full fridge at large. Modern generative systems are capable of generating plausible images of the real world and can be conditioned on the environment in which the robot operates. Here we investigate the idea of using the implicit knowledge about the world of modern generative AI systems given by their ability to generate convincing images of the real world to solve under-specified tasks.

Figures (10)

• Figure 1: Two different versions of how to stack the ingredients of a sandwich, where version (a) would likely be seen as the correct version while (b) would be seen as at least unconventional (generated with FLUX.1[dev])
• Figure 2: Generated images of different household scenarios that a robot could encounter while navigating a household and potentially performing different tasks (generated with FLUX.1[dev])
• Figure 3: Example images of dining rooms with different layouts and painting positions (generated with FLUX.1[dev])
• Figure 4: Generating images with the same layout from a base image using edge maps. (a) Base image for generating the edge map (generated with FLUX.1[dev]), (b) Edge map from the base image, (c) Image generated using the layout from the edge map (generated with FLUX.1 Canny[dev])
• Figure 5: Overview of the overall system where a camera view of a robot is used in conjunction with a text-prompt to generate an imagined version of reality where the goal state is already achieved, here for hanging a painting on the wall (images generated with FLUX.1[dev] and FLUX.1 Canny[dev])