LLM-Drone: Aerial Additive Manufacturing with Drones Planned Using Large Language Models

TL;DR

The paper tackles the challenge of performing additive manufacturing in constrained or remote environments by integrating large language models with aerial robotics. It proposes an LLM-Drone pipeline comprising an LLM Planning Module, Computer Vision Module, and a magnetic-block Mechanical Module to translate semantic design prompts into executable drone actions using magnetically connected blocks. The approach leverages coordinate synchronization via AprilTags, YOLO-based pickup verification, and reprompting for on-site error recovery, achieving substantial build accuracy and demonstrating robust LLM-driven replanning capabilities. The findings suggest that LLM-driven planning can significantly enhance on-site construction with drones, enabling flexible, scalable, and adaptive aerial manufacturing suitable for remote construction, warehouse automation, and emergency deployment. Future work envisions multilayer constructs, larger drones, and on/off magnet mechanisms to extend capability and precision.$

Abstract

Additive manufacturing (AM) has transformed the production landscape by enabling the precision creation of complex geometries. However, AM faces limitations when applied to challenging environments, such as elevated surfaces and remote locations. Aerial additive manufacturing, facilitated by drones, presents a solution to these challenges. However, despite advances in methods for the planning, control, and localization of drones, the accuracy of these methods is insufficient to run traditional feedforward extrusion-based additive manufacturing processes (such as Fused Deposition Manufacturing). Recently, the emergence of LLMs has revolutionized various fields by introducing advanced semantic reasoning and real-time planning capabilities. This paper proposes the integration of LLMs with aerial additive manufacturing to assist with the planning and execution of construction tasks, granting greater flexibility and enabling a feed-back based design and construction system. Using the semantic understanding and adaptability of LLMs, we can overcome the limitations of drone based systems by dynamically generating and adapting building plans on site, ensuring efficient and accurate construction even in constrained environments. Our system is able to design and build structures given only a semantic prompt and has shown success in understanding the spatial environment despite tight planning constraints. Our method's feedback system enables replanning using the LLM if the manufacturing process encounters unforeseen errors, without requiring complicated heuristics or evaluation functions. Combining the semantic planning with automatic error correction, our system achieved a 90% build accuracy, converting simple text prompts to build structures.

Figures (7)

• Figure 1: System Overview. (a) The main modules required for the additive manufacturing process. (b) A prompt is created from a default prompt template that includes current scene info and a design request. The LLM processes the prompt and outputs the coordinates needed to achieve the design. (c) The vision module aligns the Crazyflie coordinates with the LLM output coordinates using the Crazyflie lighthouse. (d) The drone places a block and the vision model verifies the placement. If incorrect, the current scene is passed back to (b) for remprompt and a new set of coordinates is generated to finish the design. (e) The Crazyflie drone transporting a building block from pickup to drop off.
• Figure 2: The prompt is broken into 5 parts: Design request, JSON Schema, Rules, Current Scene, and Task. The Task, Rules, and JSON Schema are predefined and do not change. The Design Request is input by the user at the start of the build process. The Current Scene is captured and presented each time a prompt is called.
• Figure 3: (a) Coordinate Sync algorithm overview. 1) Locate the origin AprilTag and the pad AprilTags. 2) Find the relative transformation between the origin and pad AprilTags. 3) Calculate the pose of the lower corner of the pad wrt. the origin. 4) Interpolate any pad coordinate by adding known vector from origin to bottom left position of pad. (b) The vision module's 3 main purposes. Pickup Detector: Green points represent corner points for Lucas-Kanade to follow. If the average of all tracking points within the YOLO bounding box is greater than the threshold (as shown on the right), the block is deemed as moved. Dropoff Detector: A frame is captured before the drone enters the region and drops a block. Background subtraction is used to compare with the initial image to determine if a block has been placed. Current State Detector: The corners of the pad are interpolated from the location of the AprilTags. The center of spatial subtraction is given by the green dot. The dot is interpolated to the closest pad location in x and y using AprilTags.
• Figure 4: (a) Model of Crazyflie pickup apparatus. A rigid tube keeps the z-length of ferrous wire constant over numerous pickup/dropoff attempts. The wire has the ability to move toward magnetic attraction from the 'pickup magnet' on the block. (b) Timelapse of the pickup and dropoff procedure. The stronger dropoff magnet allows the drone to detach from the building block once it is placed.
• Figure 5: Performance of LLMs in the quantitative test (a and b) and qualitative test (c).