Keystroke: real-time AI visuals from the piano

Idea

In 2014 my bachelor’s thesis “Synesthesy” painted images from piano harmony in real time. Keystroke is the new version, written from scratch in Python, with today’s AI tools.

How it works

1. Listening: MIDI from the piano goes into a small neural network that detects key and harmonic function.
2. Depth: A GPU renderer (moderngl, signed distance fields) turns the notes into a depth map at 30 fps.
3. Painting: For the stage I pre-render image banks with ComfyUI. Each bank has 18 layers, assigned to the harmonic functions: tonic, subdominant, dominant and their relatives.
4. Performance: Live, the depth map works like an aperture. In silence the picture only glimmers. Playing opens it by depth. Harmony picks the layer, the number of notes sets the breadth, loudness sets the brightness. Optical flow keeps the motion smooth at 40 fps.

An iPhone films the hands. OBS cuts between picture and camera, in 16:9 or 9:16.

Result

The first piece, “What recurs, what wears”, was submitted to AI Art Magazine.