Lang2Lift: A Language-Guided Autonomous Forklift System for Outdoor Industrial Pallet Handling

TL;DR

Lang2Lift is presented, an end-to-end language-guided autonomous forklift system designed to support practical pallet pick-up operations in real-world outdoor settings and provides insights into integrating language-guided perception within industrial automation systems.

Abstract

Automating pallet handling in outdoor logistics and construction environments remains challenging due to unstructured scenes, variable pallet configurations, and changing environmental conditions. In this paper, we present Lang2Lift, an end-to-end language-guided autonomous forklift system designed to support practical pallet pick-up operations in real-world outdoor settings. The system enables operators to specify target pallets using natural language instructions, allowing flexible selection among multiple pallets with different loads and spatial arrangements. Lang2Lift integrates foundation-model-based perception modules with motion planning and control in a closed-loop autonomy pipeline. Language-grounded visual perception is used to identify and segment target pallets, followed by 6D pose estimation and geometric refinement to generate manipulation-feasible insertion poses. The resulting pose estimates are directly coupled with the forklift planning and control modules to execute fully autonomous pallet pick-up maneuvers. We deploy and evaluate the proposed system on the ADAPT autonomous outdoor forklift platform across diverse real-world scenarios, including cluttered scenes, variable lighting, and different payload configurations. Tolerance-based pose evaluation further indicates accuracy sufficient for successful fork insertion. Timing and failure analyses highlight key deployment trade-offs and practical limitations, providing insights into integrating language-guided perception within industrial automation systems. Video demonstrations are available at https://eric-nguyen1402.github.io/lang2lift.github.io/

Figures (7)

• Figure 1: The ADAPT autonomous outdoor forklift equipped with our Lang2Lift framework operating in outdoor conditions.
• Figure 2: The Lang2Lift perception pipeline for automated pallet handling operations. The system processes natural language commands through Florence-2 for grounded object detection, applies SAM-2 for precise segmentation, and utilizes FoundationPose for 6D pose estimation with geometric refinement for optimal fork insertion positioning.
• Figure 3: Pose transformation process showing: (a) initial pose detection with pallet symmetry creating two possible orientations, (b) target reference position for optimal fork insertion, and (c) alternative symmetric orientation requiring correction.
• Figure 4: Representative examples of successful pallet segmentation across diverse scenarios with corresponding natural language instructions, demonstrating robust performance under varying lighting conditions, load configurations, and spatial arrangements.
• Figure 5: The visualization of vision-language object segmentation within different conditions.