Hugging Face's logo

Spaces:
Solshine
/
hyperspace-jam-lyria
Sleeping

App Files Community
hyperspace-jam-lyria / README.md
Solshine's picture
Solshine
Upload folder using huggingface_hub
33c911a verified
preview code
|
Raw
History Blame Contribute Delete
11.8 kB
metadata 
license: mit
title: Hyperspace Jam — Lyria AI Edition
sdk: docker
emoji: 🎹
colorFrom: purple
colorTo: blue
short_description: Hand-controlled AI music DJ with hyperbolic visuals

Hyperspace Jam — Lyria AI Edition

Hand-controlled AI music generation with hyperbolic geometry visuals.

Wave your hands in front of a webcam to DJ real-time AI-generated music. Hand positions, finger spread, wrist angles, and movement speed are interpreted by a local Qwen 3.5 AI model into musical descriptions, which steer Google's Lyria RealTime API to generate a continuous stream of music that responds to your gestures.

Forked from Hyperspace Jam v2 (original by collidingScopes). The original used Tone.js client-side synthesis — this version replaces all audio generation with AI-generated music via Lyria RealTime.

How It Works 

Webcam → MediaPipe Hand Tracking → Gesture Data
                                        ↓
                              ┌─────────┴──────────┐
                              │   Python Backend    │
                              │                     │
                              │  Qwen 3.5 (0.8B)   │ ← Interprets gestures
                              │  via Ollama          │   into musical text
                              │         ↓           │   descriptions
                              │  Lyria RealTime API │ ← Generates real-time
                              │  via Google GenAI    │   48kHz stereo audio
                              │         ↓           │
                              │  PCM audio stream   │
                              └─────────┬──────────┘
                                        ↓
                           Browser Web Audio API → Speakers
                           + Three.js hyperbolic geometry visuals

Data Flow Detail

  1. MediaPipe HandLandmarker tracks 30 landmarks per hand at ~30fps in the browser
  2. game.js computes gesture features: hand height, finger spread, wrist angle, velocity, finger extensions, shape detection
  3. MusicManager.js sends gesture data to the Python backend via WebSocket every 100ms
  4. Backend does two things in parallel:
    • Direct parameter mapping: Hand height → brightness, finger spread → density, wrist tilt → brightness modulation, drum hand → BPM (sent to Lyria every 500ms, only when hands move)
    • Qwen 3.5 prompt generation: Every 3 seconds (only when gesture changes significantly), the gesture state is described in natural language and sent to Qwen 3.5 0.8B via Ollama, which outputs weighted musical text prompts
  5. Lyria RealTime receives the prompts + config and generates 2-second chunks of 48kHz stereo 16-bit PCM audio, streamed back via WebSocket
  6. Browser decodes PCM chunks and schedules seamless playback via Web Audio API AudioWorklet, with an AnalyserNode feeding the hyperbolic geometry visualizer

Gesture Controls

Musical Mapping

Gesture Effect
Hand height (Y position) HIGH = ethereal, ambient, floating pads / LOW = heavy, nasty, drippy drum & bass
Finger spread WIDE = complex, dense, layered textures / NARROW = minimal, sparse, clean
Wrist tilt Tilted = acid, filter sweep, tension, dissonance
Movement speed FAST = aggressive, driving, intense / STILL = ambient, calm, sustained
White square shape (both hands) Sub-bass drone, fat sawtooth rumble
Drum hand (2nd hand) Rhythmic, percussive, breakbeats — finger extension controls BPM (60-200)
Fist Cycles through preset mood labels
Spacebar Panic stop (1 second mute)

Groove Lock

When hands stay still, the music stays in its groove — no prompt changes are sent to Lyria. Small movements trigger gradual evolution ("keep the groove, shift subtly"). Large, dramatic movements trigger a direction change ("shift genre entirely").

Transition System

The Qwen model maintains conversation history (last 6 exchanges) to ensure smooth transitions. It builds on previous themes rather than jumping randomly between styles.

Architecture

Frontend (Browser)

File Purpose
index.html Entry point, UI controls, import map for Three.js
main.js Bootstraps the Game instance
game.js MediaPipe hand tracking, gesture feature extraction, hand rendering, visualization orchestration. Tone.js replaced with a minimal shim.
MusicManager.js Lyria drop-in replacement for original Tone.js synth engine. Sends gesture data to backend via WebSocket, receives PCM audio, plays via Web Audio API. Maintains full interface compatibility with game.js.
DrumManager.js Stub module (Lyria generates all music including drums). Exports Map-based interface for game.js compatibility.
WaveformVisualizer.js Poincare disk hyperbolic geometry shader, audio-reactive
MandalaVisualizer.js Sacred geometry rendering with pre-allocated pools
ShapeManager.js Pinch shape tracking between hands (white quadrilateral)
ShapeTessellationShader.js GLSL hyperbolic tiling for shape fill
DisplacementFilter.js SVG turbulence + smoke DOM-based warping
styles.css Dark kiosk-style UI

Backend (Python)

File Purpose
backend/server.py FastAPI + WebSocket server. Bridges frontend gestures to Lyria + Qwen. Serves frontend static files. Implements gesture dead-zone logic (only updates on significant movement) and distinguishes gradual vs dramatic changes.
backend/lyria_manager.py Manages Lyria RealTime WebSocket session via google-genai SDK. Handles connect, play/pause/stop, prompt updates, config updates (brightness, density, BPM, mute toggles), context resets, and audio chunk streaming.
backend/gesture_interpreter.py Translates gesture data into Lyria weighted prompts via Qwen 3.5 0.8B (Ollama). Maintains conversation history for smooth transitions. Falls back to rule-based prompt generation when Ollama is unavailable.
backend/requirements.txt Python dependencies

Key Technologies

Technology Role Version/Model
Google Lyria RealTime AI music generation (streaming) models/lyria-realtime-exp via Gemini API v1alpha
Qwen 3.5 Gesture-to-prompt interpretation qwen3.5:0.8b via Ollama (1GB, ~1s inference)
MediaPipe Hand landmark detection HandLandmarker v0.10.14
Three.js WebGL visualization v0.161.0
FastAPI Backend WebSocket server Latest
Web Audio API PCM audio playback Browser native

Setup

Prerequisites

  • Python 3.10+
  • Ollama running locally with qwen3.5:0.8b model
  • Google Gemini API key with Lyria RealTime access
  • Modern browser with webcam access (Chrome recommended)

Installation 

# Clone the repository
git clone https://github.com/SolshineCode/hyperspace-jam-lyria.git
cd hyperspace-jam-lyria

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
source .venv/Scripts/activate    # Windows/Git Bash

# Install Python dependencies
pip install -r backend/requirements.txt

# Pull the Qwen model (if not already installed)
ollama pull qwen3.5:0.8b

Configuration

Local development: Create a .env file in the project root:

GEMINI_API_KEY=your-gemini-api-key-here

Hugging Face Spaces: Add GEMINI_API_KEY as a Secret in the Space settings (Settings > Repository Secrets).

The .env file is in .gitignore and will not be committed.

Running 

# Make sure Ollama is running
ollama serve  # (if not already running)

# Start the server
python -u backend/server.py

# Open in browser
# http://localhost:8088

The server starts on port 8088 by default. It serves both the API (WebSocket at /ws) and the frontend static files.

Quick Start (one command) 

bash start.sh

Lyria RealTime API Details

Connection

  • Protocol: WebSocket (persistent, bidirectional, low-latency)
  • Model: models/lyria-realtime-exp
  • SDK: google-genai with api_version='v1alpha'
  • Audio format: Raw 16-bit PCM, 48kHz, stereo
  • Chunk size: 384,000 bytes (2 seconds of audio)
  • First chunk latency: ~3-4 seconds after play()
  • Session limit: 10 minutes (auto-reconnect planned)

Controls Used

Parameter Range Mapped From
weighted_prompts Text + weight (0.1-3.0) Qwen 3.5 output from gesture interpretation
brightness 0.0-1.0 Hand height * wrist tilt factor
density 0.0-1.0 Finger spread * volume
bpm 60-200 Drum hand finger extension average (requires context reset)
mute_drums bool Gesture flag
mute_bass bool Gesture flag
temperature 0.0-3.0 Fixed at 1.1 (default)
guidance 0.0-6.0 Fixed at 4.0 (prompt adherence)

Prompt Engineering

The Qwen model receives structured gesture descriptions like:

Hand HIGH (0.8), fingers WIDE, wrist STRAIGHT, movement STILL.

Evolve gradually — keep the groove, shift subtly.

And outputs weighted prompt JSON:

{
  "prompts": [
    {"text": "ethereal ambient floating pads celestial", "weight": 1.5},
    {"text": "shimmering reverb atmosphere", "weight": 1.0}
  ]
}

The conversation history ensures coherent transitions between styles.

Gesture Change Detection

The backend implements a two-tier change detection system:

Significant Change (triggers config update + Qwen prompt)

  • Hand height moved > 0.12
  • Finger spread changed > 0.15
  • Wrist angle changed > 0.25
  • Hand velocity > 0.25
  • Shape (white square) toggled
  • Drum hand toggled

Major Change (triggers Qwen "change direction" mode)

  • Hand height moved > 0.3
  • Finger spread changed > 0.35
  • Hand velocity > 0.5
  • Shape toggled

If neither threshold is met, no updates are sent — the music stays locked in its current groove.

Deployment to Hugging Face Spaces

  1. Create a private Space on Hugging Face (SDK: Docker or Static)
  2. Add GEMINI_API_KEY as a Secret in Space settings
  3. Push the repository to the Space
  4. The backend will read the API key from environment variables

Known Limitations

  • Lyria session limit: 10 minutes max per session. Page refresh starts a new session.
  • Audio latency: ~3-4 second initial latency for first chunk, then ~2 second chunks arrive smoothly. Not suitable for rhythm-game-level precision.
  • Qwen inference: ~1-2 seconds per prompt generation on CPU with 0.8B model. Doesn't block audio streaming.
  • No vocals: Lyria RealTime generates instrumental music only.
  • Experimental API: Lyria RealTime is in v1alpha — API may change.

Credits

  • Original Hyperspace Jam: collidingScopes — hand tracking + Tone.js synth + hyperbolic visuals
  • Lyria RealTime: Google DeepMind — real-time AI music generation
  • Qwen 3.5: Alibaba/Qwen — small language model for gesture interpretation
  • MediaPipe: Google — hand landmark detection
  • Three.js: mrdoob — WebGL rendering
  • Ollama: ollama.com — local LLM inference

License

MIT License