Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Real-Time Gesture-Based Control Framework

Mahya Khazaei, Ali Bahrani, George Tzanetakis

TL;DR

This work presents a real-time gesture-based sound control framework that fuses live video-based body landmark extraction with ML-driven gesture classification to modulate audio in real time. The architecture leverages MediaPipe for landmark detection, Max/MSP for audio processing, and OSC-enabled Python components for learning and inference, achieving sub-200 ms latency. A practical training workflow enables users to label gesture samples with as few as ~50–80 examples, and a mapping mechanism translates gestures into audio controls such as tempo, pitch, gains, and playback sequencing. Demonstrations in dance and real-time sound control scenarios reveal robust performance, competitive accuracy relative to established gesture-recognition baselines, and clear pathways for extending to therapy, education, and immersive installations.

Abstract

We introduce a real-time, human-in-the-loop gesture control framework that can dynamically adapt audio and music based on human movement by analyzing live video input. By creating a responsive connection between visual and auditory stimuli, this system enables dancers and performers to not only respond to music but also influence it through their movements. Designed for live performances, interactive installations, and personal use, it offers an immersive experience where users can shape the music in real time. The framework integrates computer vision and machine learning techniques to track and interpret motion, allowing users to manipulate audio elements such as tempo, pitch, effects, and playback sequence. With ongoing training, it achieves user-independent functionality, requiring as few as 50 to 80 samples to label simple gestures. This framework combines gesture training, cue mapping, and audio manipulation to create a dynamic, interactive experience. Gestures are interpreted as input signals, mapped to sound control commands, and used to naturally adjust music elements, showcasing the seamless interplay between human interaction and machine response.

A Real-Time Gesture-Based Control Framework

TL;DR

This work presents a real-time gesture-based sound control framework that fuses live video-based body landmark extraction with ML-driven gesture classification to modulate audio in real time. The architecture leverages MediaPipe for landmark detection, Max/MSP for audio processing, and OSC-enabled Python components for learning and inference, achieving sub-200 ms latency. A practical training workflow enables users to label gesture samples with as few as ~50–80 examples, and a mapping mechanism translates gestures into audio controls such as tempo, pitch, gains, and playback sequencing. Demonstrations in dance and real-time sound control scenarios reveal robust performance, competitive accuracy relative to established gesture-recognition baselines, and clear pathways for extending to therapy, education, and immersive installations.

Abstract

We introduce a real-time, human-in-the-loop gesture control framework that can dynamically adapt audio and music based on human movement by analyzing live video input. By creating a responsive connection between visual and auditory stimuli, this system enables dancers and performers to not only respond to music but also influence it through their movements. Designed for live performances, interactive installations, and personal use, it offers an immersive experience where users can shape the music in real time. The framework integrates computer vision and machine learning techniques to track and interpret motion, allowing users to manipulate audio elements such as tempo, pitch, effects, and playback sequence. With ongoing training, it achieves user-independent functionality, requiring as few as 50 to 80 samples to label simple gestures. This framework combines gesture training, cue mapping, and audio manipulation to create a dynamic, interactive experience. Gestures are interpreted as input signals, mapped to sound control commands, and used to naturally adjust music elements, showcasing the seamless interplay between human interaction and machine response.
Paper Structure (29 sections, 3 figures, 6 tables)

This paper contains 29 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Gesture and Body Movement Recognition Systems
  4. Deep Learning for Gesture Recognition
  5. Ensemble-Driven Hand Pose Estimation
  6. Real-Time Gesture Recognition
  7. Real-time Sound Control Systems
  8. Gesture-based Interactive Sound Mapping systems
  9. system Description
  10. System architecture
  11. System Components
  12. Audio Manipulation
  13. Body Landmark Extraction
  14. Bridging Real-Time Communication
  15. Machine Learning Mode
  16. ...and 14 more sections

Figures (3)

  • Figure 1: System Data Flow Architecture
  • Figure 2: Mediapipe body pose landmarks and feature selection
  • Figure 3: user A learning curve