Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations

Koffivi Fidèle Gbagbe, Miguel Altamirano Cabrera, Ali Alabbas, Oussama Alyunes, Artem Lykov, Dzmitry Tsetserukou

TL;DR

Bi-VLA presents a unified vision-language-action framework to enable bimanual dexterous manipulation driven by natural language instructions. The pipeline combines a semantic LLM-based planner, a code-generation step translating plans into API calls, a motion controller, and a Qwen Vision-Language Model for scene grounding and 3D localization, with explicit API function sets to coordinate dual arms. Key contributions include a modular architecture with a defined Action API surface, robust 2D-to-3D grounding under distortion correction, and an end-to-end cooking-domain evaluation showing strong LLM and VLM performance and competitive task execution rates. The work advances human-robot interaction by reducing data requirements and facilitating flexible, instruction-driven bimanual manipulation in dynamic kitchen-like tasks, with practical implications for assistive and collaborative robotics.

Abstract

This research introduces the Bi-VLA (Vision-Language-Action) model, a novel system designed for bimanual robotic dexterous manipulation that seamlessly integrates vision for scene understanding, language comprehension for translating human instructions into executable code, and physical action generation. We evaluated the system's functionality through a series of household tasks, including the preparation of a desired salad upon human request. Bi-VLA demonstrates the ability to interpret complex human instructions, perceive and understand the visual context of ingredients, and execute precise bimanual actions to prepare the requested salad. We assessed the system's performance in terms of accuracy, efficiency, and adaptability to different salad recipes and human preferences through a series of experiments. Our results show a 100% success rate in generating the correct executable code by the Language Module, a 96.06% success rate in detecting specific ingredients by the Vision Module, and an overall success rate of 83.4% in correctly executing user-requested tasks.

Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations

TL;DR

Bi-VLA presents a unified vision-language-action framework to enable bimanual dexterous manipulation driven by natural language instructions. The pipeline combines a semantic LLM-based planner, a code-generation step translating plans into API calls, a motion controller, and a Qwen Vision-Language Model for scene grounding and 3D localization, with explicit API function sets to coordinate dual arms. Key contributions include a modular architecture with a defined Action API surface, robust 2D-to-3D grounding under distortion correction, and an end-to-end cooking-domain evaluation showing strong LLM and VLM performance and competitive task execution rates. The work advances human-robot interaction by reducing data requirements and facilitating flexible, instruction-driven bimanual manipulation in dynamic kitchen-like tasks, with practical implications for assistive and collaborative robotics.

Abstract

This research introduces the Bi-VLA (Vision-Language-Action) model, a novel system designed for bimanual robotic dexterous manipulation that seamlessly integrates vision for scene understanding, language comprehension for translating human instructions into executable code, and physical action generation. We evaluated the system's functionality through a series of household tasks, including the preparation of a desired salad upon human request. Bi-VLA demonstrates the ability to interpret complex human instructions, perceive and understand the visual context of ingredients, and execute precise bimanual actions to prepare the requested salad. We assessed the system's performance in terms of accuracy, efficiency, and adaptability to different salad recipes and human preferences through a series of experiments. Our results show a 100% success rate in generating the correct executable code by the Language Module, a 96.06% success rate in detecting specific ingredients by the Vision Module, and an overall success rate of 83.4% in correctly executing user-requested tasks.
Paper Structure (17 sections, 4 equations, 4 figures, 1 table)

This paper contains 17 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. From language instructions to robotic actions
  4. Bimanual robotic manipulation
  5. General system overview
  6. Large Language Model Module
  7. Semantic planning
  8. Translating semantic plan into APIs function calls with the Code Generator
  9. From API function call to Action Generation with The Motion Control APIs
  10. Vision Language Model Module
  11. Integration of the Qwen Vision-Language Model
  12. Mapping Image Pixel to 3D Coordinates
  13. Experimental Evaluation
  14. Experimental Setup
  15. Experiments Workflow
  16. ...and 2 more sections

Figures (4)

  • Figure 1: System Overview of Bi-VLA.
  • Figure 2: Bi-VLA Architecture.
  • Figure 3: Cooking Experiment Workflow.
  • Figure 4: Example illustration of Action 2 in Fig. \ref{['experiment']}. The LLM-based semantic planner receives the action to perform and generates a detailed plan outlining the movements of two robot manipulators. The generated plan is translated into a set of motion API call functions through the LLM-based Code Generator. The execution of the generated code triggers the movement of the two robot manipulators through the Motion Controller.(a)-(b) the manipulator with the gripper moves to grasp the pepper and bring it to the cutting board; (c)-(d) the manipulator with the knife moves to the cutting board, cuts the pepper (e), and places it in the bowl; (f)-(g) the two manipulators return to their initial position.