Archive 2020: Contactless Interface: Depth Sensor to MIDI for Real-Time Sound Control

A contactless instrument interface that translates hand movement into real-time sound manipulation. An Intel RealSense depth sensor tracks hand position in three-dimensional space, and that spatial data drives the cutoff frequency of a synthesizer running inside Native Instruments Maschine — no physical contact required.

The signal chain bridges three protocols. TouchDesigner reads the raw depth stream from the Intel sensor and maps hand distance to a normalised control value. That value is sent as an OSC message to OSCulator, which converts it into MIDI continuous controller data. Maschine receives the MIDI input and applies it to the filter cutoff of the active sound in real time. Moving a hand closer to the sensor opens the filter; pulling away closes it — an intuitive, physical gesture mapped to a sonic parameter.

Why TouchDesigner

TouchDesigner acts as the central nervous system of the setup. It handles depth image processing, gesture extraction, value smoothing, and OSC output in a single visual programming environment. Its real-time architecture makes it well suited for low-latency control chains where perceptible delay between gesture and sound would break the musical connection.

Visual feedback

The external display renders a real-time visualisation of the cutoff frequency directly from TouchDesigner — a generative line pattern that deforms in response to the same depth data driving the audio. The visual and sonic layers respond to the same gesture simultaneously, closing the feedback loop between performer and instrument.

What the experiment revealed

The setup worked exactly as designed. But running it in practice exposed two things the technical spec couldn't predict.

First: without visual feedback, the interaction disappears. To anyone watching, the sound changes seem to happen on their own — there is no visible cause, no legible connection between hand and effect. The gesture that feels expressive to the performer reads as randomness to an observer.

Second: the choice of parameter matters more than the precision of the control. Filter cutoff is intimate territory for a sound designer — the relationship between hand position and timbre feels direct and meaningful. But for anyone outside that world, it reads as abstract. Had the sensor controlled volume instead, the connection would have been immediate and universal. The experiment worked technically. What it exposed was the distance between a parameter that means something to the maker and one that is legible to an audience.