Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation

Heng Zhang, Wei-Hsing Huang, Gokhan Solak, Arash Ajoudani

TL;DR

OmniVIC addresses safe, generalizable manipulation in contact-rich environments by unifying a universal variable impedance controller with a vision-language model, retrieval-augmented generation, and in-context learning. The VLM grounds task context to generate adaptive impedance parameters, while RAG retrieves relevant past experiences and ICL contextualizes them to produce real-time gains for $K$ and $D$, all regulated by force/torque feedback. Across extensive simulations and real-robot experiments, OmniVIC significantly improves task success (61.4% vs 27% baseline) and reduces force violations, demonstrating safer, more adaptable manipulation. By bridging high-level semantic reasoning with low-level compliant control, the method offers a scalable path toward robust manipulation in diverse, unstructured settings.

Abstract

We present OmniVIC, a universal variable impedance controller (VIC) enhanced by a vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. Traditional VIC have shown advantages when the robot physically interacts with the environment, but lack generalization in unseen, complex, and unstructured safe interactions in universal task scenarios involving contact or uncertainty. To this end, the proposed OmniVIC interprets task context derived reasoning from images and natural language and generates adaptive impedance parameters for a VIC controller. Specifically, the core of OmniVIC is a self-improving Retrieval-Augmented Generation(RAG) and in-context learning (ICL), where RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks, and ICL leverages these retrieved examples and the prompt of current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario. Therefore, a self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios. The impedance parameter regulation is further informed by real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms baselines on a suite of complex contact-rich tasks, both in simulation and on real-world robotic tasks, with improved success rates and reduced force violations. OmniVIC takes a step towards bridging high-level semantic reasoning and low-level compliant control, enabling safer and more generalizable manipulation. Overall, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC).

OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation

TL;DR

OmniVIC addresses safe, generalizable manipulation in contact-rich environments by unifying a universal variable impedance controller with a vision-language model, retrieval-augmented generation, and in-context learning. The VLM grounds task context to generate adaptive impedance parameters, while RAG retrieves relevant past experiences and ICL contextualizes them to produce real-time gains for and , all regulated by force/torque feedback. Across extensive simulations and real-robot experiments, OmniVIC significantly improves task success (61.4% vs 27% baseline) and reduces force violations, demonstrating safer, more adaptable manipulation. By bridging high-level semantic reasoning with low-level compliant control, the method offers a scalable path toward robust manipulation in diverse, unstructured settings.

Abstract

We present OmniVIC, a universal variable impedance controller (VIC) enhanced by a vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. Traditional VIC have shown advantages when the robot physically interacts with the environment, but lack generalization in unseen, complex, and unstructured safe interactions in universal task scenarios involving contact or uncertainty. To this end, the proposed OmniVIC interprets task context derived reasoning from images and natural language and generates adaptive impedance parameters for a VIC controller. Specifically, the core of OmniVIC is a self-improving Retrieval-Augmented Generation(RAG) and in-context learning (ICL), where RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks, and ICL leverages these retrieved examples and the prompt of current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario. Therefore, a self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios. The impedance parameter regulation is further informed by real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms baselines on a suite of complex contact-rich tasks, both in simulation and on real-world robotic tasks, with improved success rates and reduced force violations. OmniVIC takes a step towards bridging high-level semantic reasoning and low-level compliant control, enabling safer and more generalizable manipulation. Overall, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC).
Paper Structure (31 sections, 2 equations, 7 figures, 1 table)

This paper contains 31 sections, 2 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Variable impedance control for physical tasks
  4. Retrieval-Augmented generation for robotics
  5. In-context learning
  6. Leveraging VLMs for VIC controller
  7. Methodology
  8. VLM-informed VIC
  9. RAG data acquisition and storage
  10. RAG retrieval
  11. Step 1: instruction filtering
  12. Step 2: phase filtering
  13. Step 3: similarity computation
  14. Step 4: final score
  15. ICL‑enhanced impedance parameters prediction
  16. ...and 16 more sections

Figures (7)

  • Figure 1: Overview of the proposed OmniVIC. The system integrates a VLM with a VIC to enable safe and adaptive manipulation in contact-rich tasks. The VLM processes multimodal inputs, including visual observations, natural language instructions, and real-time force/torque feedback, to generate context-aware impedance parameters (stiffness and damping) for the VIC.
  • Figure 2: Tasks used in simulation experiments.
  • Figure 3: Deep dive into force and impedance profiles during task execution (Task 7 in Table \ref{['tab:whole_task']}) in low-level controller. Left and middle: stiffness and damping profiles of OmniVIC. Right: force (only $F_z$, the main contact force) profiles. The impedance profiles show that OmniVIC dynamically adjusts stiffness and damping based on task phases, reducing them during contact (middle area) to enhance compliance. The blue dots in force profiles demonstrate that OmniVIC maintains interaction forces within safe limits, avoiding spikes that violate constrain, while it happens (yellow dots in the red box) in the baseline controller and terminates the episode as a failure.
  • Figure 4: Simulation results: Task success rates of baseline ($\pi_0$ with position control) vs. OmniVIC (with VLM-informed variable impedance control).
  • Figure 5: Real-world experiments: Top: Task 1: The robot gently closes the top drawer. Middle: Task 2: The robot moves along the negative Y-axis while maintaining the same height in Z-axis, where there is a ramp in the middle of the path to introduce contact. Bottom: Task 3: the same as Task 2, but with two different ramp shapes to test multiple phases of contact.
  • ...and 2 more figures