Research questions

1. How can movement captured without wearable sensors become playable, closer to bowing a string than to pressing a trigger?

2. What changes when a physical, moving object enters the feedback loop between body and sound?

3. Where should a machine performer sit on the spectrum between responsiveness and autonomy?

4. How can a system’s interpretation of the body be made legible to performers and audiences?

A proposed performance ecosystem

The conceptual configuration we arrived at places a human performer between two robots in a shared interaction area.

• • Camera system: observes both the human and the machines.
  • Projected avatar: shown on a semi-transparent gauze screen, it displays the system’s reading of the performer. It answers the question how does the system see the human?
  • Consoles: two placed at the edges of the space show the running code, offering a view behind the curtain.
  • Robots: respond to the performer through their own programmed logic. They can move with the human or against them, in the manner of voices in a polyphonic counterpoint: in parallel, in contrary motion, or switching between the two

To think through how such a performance could be composed, we drafted a hybrid score with three layers. The performer layer notates gestures and postures. The oscillator layer describes synthesis states. The system layer notates the machine’s logic: voice-leading states, discrete decision points such as programmed pauses or reactive moments, and interference points where competing logics overlap and produce emergent behaviour. The score treats the computational system as a performer with its own notated part.

Technical overview

Pose estimation. The system uses a standard webcam feed and the BlazePose model, a convolutional neural network for body tracking, via Google’s MediaPipe library in Python. It detects and continuously tracks a set of body keypoints (joints and other anatomical landmarks) without markers or wearable sensors.

Interaction layers. Raw keypoint data is refined through transfer functions into parameters for sound and motion. We structured these mappings in three orders:

1st Order: Absolute positions of participants’ body joints.

• • Cartesian coordinate mapping
  • Position/depth perception using AI [Python MediaPipe library, BlazePose CNN]

2nd Order: Relative positions between participant joints and robot joints. Defining ratios and equations:

• • Hips to arms
  • Legs to hands
  • Arms to torso

3rd Order: Derived values and abstract relationships.

• • Differences in speed, delay between movements, reactivity
  • State tracking and memory implementation
    → f(x, y) → |(p_x + p_y)| + |(d_x − d_y)|
  • Control structures and decision-making trees

Sound. The mapped data drives real-time synthesis in SuperCollider. Small, quick gestures such as a flick of the wrist produce granular clusters. Slower shifts in posture modulate the timbre of sustained textures. The sound material ranges from sine-wave banks and tuned resonators to granulation and multi-resonator feedback networks.

Motion. Each robot arm is driven by its own DYNAMIXEL actuator, two per robot. A wireless ESP8266 microcontroller connects each robot to a central computer, which coordinates sound synthesis and robotic movement within one control loop.

Feedback. The projected avatar closes the loop. Performers can see how the system interprets their movement and adjust, much as a violinist adjusts bow pressure in response to the sound.

Prototype

Alongside the software pipeline, we built a motorised prototype to test the translation from tracked movement to physical gesture. The technical demo shows the pose-estimation pipeline and the prototype in operation.

Reflection

The full installation was not realised. The research left us with a working framework for camera-based movement sonification and a tested link between tracked bodies and motorised objects. It also left a set of open questions about causality and presence: what it means for a sound to appear to come from a moving object, and how far a machine’s movement can carry musical intention. The camera-based approach developed here later fed directly into Music for Seeing Circuits, where a live computer vision system reads a score in real time during performance.