linkedin-post.md

I built a thing where you DJ with your hands.

Webcam. Hand tracking. AI music that changes based on where your hands are and what your fingers are doing. Hands up high and it's ambient psybient. Drop them low and you get this filthy, drippy neurofunk that honestly surprised me with how good it sounds. Each finger controls a different layer. Both fists closed = bass drop that locks out all other inputs until you release.

You can try it right now — just a browser and a webcam, nothing to install. The Lyria API is free for sessions up to 10 minutes: https://huggingface.co/spaces/Solshine/hyperspace-jam-lyria

This runs on @Google DeepMind's Lyria RealTime API, which is different from their standard Lyria in a way that matters for developers. Standard Lyria is a batch endpoint — send a prompt, get a 30-second WAV back. RealTime opens a persistent WebSocket and streams continuous 48kHz stereo that you can steer with updated prompts and parameters while it plays. It's closer to a MIDI instrument that plays itself from your text descriptions than it is to a clip generator. The model generates in 2-second chunks internally, looking backward at what it just played to keep the rhythm locked while looking forward at your latest controls. So you can slide from ambient chillout to heavy DnB and the kick pattern stays coherent, the key doesn't jump randomly, the energy ramps instead of cutting. I morphed through five genres in one 10-minute session and it felt like one continuous mix, not five clips stitched together. That continuity is the whole point for live performance — you need the music to feel like it's evolving, not resetting.

The architecture might interest people building with multiple AI services: I chain three models together in a real-time loop.

@Google's MediaPipe does hand tracking in the browser — 30 landmarks per hand at 30fps, no server round-trip. That hand data splits into two paths. Path 1: numeric features (hand height, finger spread, wrist tilt) map directly to Lyria's physics parameters (brightness, density, BPM) through simple math. Fast, every 1.5 seconds. Path 2: a local @Alibaba Cloud Qwen 3.5 model running through @Ollama (0.8B params, 1GB, no GPU needed) translates the gesture state into creative weighted text prompts for Lyria — like "squishy neuro bass granular stutter" at weight 2.0. Both paths feed Lyria simultaneously. Physics controls HOW it sounds. Text controls WHAT it sounds like.

The Qwen-through-Ollama part is worth talking about for anyone building on top of music/audio APIs. You can't just send raw numbers to Lyria and get good results. "Hand height 0.3" doesn't mean anything musically. You need something that turns coordinates into creative musical descriptions — and varies them so you don't hear the same thing for 10 minutes. A 0.8B LLM running locally does this for $0 inference cost and ~1 second latency. Rules-based mapping gives you the same output every time. The LLM says "squishy granular neuro growl" one time and "distorted FM bass with tape saturation" the next. Over a session that variety is what makes it feel alive instead of robotic.

Same hand data that feeds the music also drives psychedelic hyperbolic geometry visuals at 30fps. The white square you can form between your two hands is both a visual tessellation AND a sub-bass trigger in Lyria at the same time. One input, two outputs, zero coupling between them.

For the EDM producers and sound designers here — some Lyria RealTime tricks I found:

• Don't describe physics in text prompts. Lyria has dedicated API params for brightness (0.0-1.0) and density (0.0-1.0). Putting "bright" or "dense" in your text just conflicts with the params. Text = genre and instruments only.
• DIVERSITY mode or don't bother for electronic music. QUALITY mode is safe and boring. DIVERSITY pushes weird timbres, unexpected sound design, the kind of stuff you actually want in psytrance or neuro.
• BPM 172 + "half-time groove" prompt gives you this bouncy neurohop feel at perceived 86 BPM but with fast DnB fills in the gaps. Accidentally found this and it's my favorite zone in the whole app.
• Crossfade your prompts. Don't slam new prompts in — fade old ones down while new ones come up. One intermediate blend step. The genre transitions go from jarring to smooth.
• When nothing's changing, send NOTHING. Let Lyria ride. The groove needs time to breathe. This was the hardest lesson — my engineer brain wanted to keep the system responsive but for music, silence from the controller is the feature.

It's all open source. Fork it, remix it, build on it: https://github.com/SolshineCode/hyperspace-jam-lyria

Built with @Anthropic's Claude Code in one session. About 3,300 lines across 17 files, 74 commits. The whole inference stack besides Lyria API calls is $0 — @Ollama runs Qwen locally, @Google MediaPipe runs in-browser.

What I actually want to do with this: put it in a geodesic dome at a festival. Cameras, big screen, hyperbolic geometry breathing on the walls. People walk up and start making music with their bodies. Two people who don't speak the same language creating something together without saying a word. I don't know exactly how to make that happen yet but I'm going to figure it out. If you're into festival tech or interactive installations, let's talk.

Huge thanks to the @Google DeepMind Lyria team for putting this API out there. Real-time steerable music generation over WebSocket is not something I've found anywhere else. And to the @Qwen team at @Alibaba Cloud and the @Ollama team — the idea that a 0.8B model on a laptop with no GPU can be the creative brain feeding a cloud music generation API is still kind of blowing my mind.

#GoogleLyria #LyriaRealTime #GoogleDeepMind #AIMusic #HandTracking #MediaPipe #OpenSource #ElectronicMusic #DnB #Neurofunk #PsychedelicMusic #CreativeAI #MusicTech #Ollama #Qwen #GenerativeAI #WebAudio #Psytrance #SoundDesign #MusicProduction #FestivalTech