NeoLightning: A Modern Reimagination of Gesture-Based Sound Design

TL;DR

NeoLightning tackles the inaccessibility of the Buchla Lightning by reimplementing gestural control with modern tools. The authors combine MediaPipe hand landmarks with a Max/MSP audio path and a Processing-based visualization to deliver precise, low-latency, camera-based gesture interaction and a 3D interface for immersive sound design. They map gesture axes to pitch, loudness, and timbre with additional controls via sphere size, while integrating P_4L for emergent textures; a complementary keyboard mode ensures accessibility without cameras. The work demonstrates a reproducible, extensible approach to reimagining historical DMIs, with potential for spatial audio, mobile, and AR contexts.

Abstract

This paper introduces NeoLightning, a modern reinterpretation of the Buchla Lightning. NeoLightning preserves the innovative spirit of Don Buchla's "Buchla Lightning" (introduced in the 1990s) while making its gesture-based interaction accessible to contemporary users. While the original Buchla Lightning and many other historical instruments were groundbreaking in their time, they are now largely unsupported, limiting user interaction to indirect experiences. To address this, NeoLightning leverages MediaPipe for deep learning-based gesture recognition and employs Max/MSP and Processing for real-time multimedia processing. The redesigned system offers precise, low-latency gesture recognition and immersive 3D interaction. By merging the creative spirit of the original Lightning with modern advancements, NeoLightning redefines gesture-based musical interaction, expanding possibilities for expressive performance and interactive sound design.

Figures (3)

• Figure 1: Buchla Lightning and Gesture-Based Instruments (Source matrixsynth2014buchla)
• Figure 2: System workflow integrating gesture, sound, and visualization
• Figure 3: System architecture of NeoLightning