GAgent: An Adaptive Rigid-Soft Gripping Agent with Vision Language Models for Complex Lighting Environments

TL;DR

This work tackles open-world UAV grasping under variable lighting by introducing GAgent, a VLM-guided soft gripping framework with a brightness-enhancement tool. It integrates a Prompt Engineer, a Visual-Language Model core, and a Workflow to translate perception into grasp planning, augmented by a low-light enhancement strategy. A novel bionic hybrid gripper combines rigid springs, silicone, cables, and a convergent tendon layout, enabling variable stiffness characterized by a stiffness metric $k_T = \frac{k_s k_t k_m}{k_t k+m + k_s k_m + k_s k_t}$ and fingertip deflection expressions such as $\delta_{tip}$ and $\theta_L = \frac{F L^2}{2(EI + r^2 L k_t)}$. Through FE analyses and multiple experiments, the study shows that convergent tendons improve stiffness and load bearing, pretensioning accelerates stabilization, and the integrated VLM with brightness adjustment achieves robust grasps across diverse objects and lighting, highlighting potential UAV deployment in challenging environments.

Abstract

This paper introduces GAgent: an Gripping Agent designed for open-world environments that provides advanced cognitive abilities via VLM agents and flexible grasping abilities with variable stiffness soft grippers. GAgent comprises three primary components - Prompt Engineer module, Visual-Language Model (VLM) core and Workflow module. These three modules enhance gripper success rates by recognizing objects and materials and accurately estimating grasp area even under challenging lighting conditions. As part of creativity, researchers also created a bionic hybrid soft gripper with variable stiffness capable of gripping heavy loads while still gently engaging objects. This intelligent agent, featuring VLM-based cognitive processing with bionic design, shows promise as it could potentially benefit UAVs in various scenarios.

Figures (12)

• Figure 1: LGAgent (Lowlight Gripping Agent) consists of three key components: the Prompt Engineer, the Visual-Language Model (VLM), and the Work Flow.
• Figure 2: Anatomy of the human hand
• Figure 3: Three-finger hybrid gripper construction
• Figure 4: Simplified schematic. (a) Simplified diagram of the gripper as a whole, (b) forces on the fingertips.
• Figure 5: Comparison of fingertip stiffness of pure soft material, soft material with parallel tendon, and soft material with convergent tendon. The horizontal coordinate represents the end position of the tendon mount and the vertical coordinate represents the deflection of the end of the finger. The red, blue, and green lines represent fingertip deflection for pure soft material, soft material and parallel tendon, and soft material and convergent tendon, respectively.