---
license: mit
title: Hyperspace Jam — Lyria AI Edition
sdk: docker
emoji: 🎹
colorFrom: purple
colorTo: blue
short_description: Hand-controlled AI music DJ with hyperbolic visuals
---

# Hyperspace Jam — Lyria AI Edition

**Hand-controlled AI music generation with hyperbolic geometry visuals.**

Wave your hands in front of a webcam to DJ real-time AI-generated music. Hand positions, finger spread, wrist angles, and movement speed are interpreted by a local Qwen 3.5 AI model into musical descriptions, which steer Google's Lyria RealTime API to generate a continuous stream of music that responds to your gestures.

> Forked from [Hyperspace Jam v2](https://github.com/SolshineCode/hyperspace-jam-v2) (original by [collidingScopes](https://github.com/collidingScopes/arpeggiator)). The original used Tone.js client-side synthesis — this version replaces all audio generation with AI-generated music via Lyria RealTime.

---

## How It Works

```
Webcam → MediaPipe Hand Tracking → Gesture Data
                                        ↓
                              ┌─────────┴──────────┐
                              │   Python Backend    │
                              │                     │
                              │  Qwen 3.5 (0.8B)   │ ← Interprets gestures
                              │  via Ollama          │   into musical text
                              │         ↓           │   descriptions
                              │  Lyria RealTime API │ ← Generates real-time
                              │  via Google GenAI    │   48kHz stereo audio
                              │         ↓           │
                              │  PCM audio stream   │
                              └─────────┬──────────┘
                                        ↓
                           Browser Web Audio API → Speakers
                           + Three.js hyperbolic geometry visuals
```

### Data Flow Detail

1. **MediaPipe HandLandmarker** tracks 30 landmarks per hand at ~30fps in the browser
2. **game.js** computes gesture features: hand height, finger spread, wrist angle, velocity, finger extensions, shape detection
3. **MusicManager.js** sends gesture data to the Python backend via WebSocket every 100ms
4. **Backend** does two things in parallel:
   - **Direct parameter mapping**: Hand height → brightness, finger spread → density, wrist tilt → brightness modulation, drum hand → BPM (sent to Lyria every 500ms, only when hands move)
   - **Qwen 3.5 prompt generation**: Every 3 seconds (only when gesture changes significantly), the gesture state is described in natural language and sent to Qwen 3.5 0.8B via Ollama, which outputs weighted musical text prompts
5. **Lyria RealTime** receives the prompts + config and generates 2-second chunks of 48kHz stereo 16-bit PCM audio, streamed back via WebSocket
6. **Browser** decodes PCM chunks and schedules seamless playback via Web Audio API AudioWorklet, with an AnalyserNode feeding the hyperbolic geometry visualizer

---

## Gesture Controls

### Musical Mapping

| Gesture | Effect |
|---------|--------|
| **Hand height (Y position)** | HIGH = ethereal, ambient, floating pads / LOW = heavy, nasty, drippy drum & bass |
| **Finger spread** | WIDE = complex, dense, layered textures / NARROW = minimal, sparse, clean |
| **Wrist tilt** | Tilted = acid, filter sweep, tension, dissonance |
| **Movement speed** | FAST = aggressive, driving, intense / STILL = ambient, calm, sustained |
| **White square shape** (both hands) | Sub-bass drone, fat sawtooth rumble |
| **Drum hand** (2nd hand) | Rhythmic, percussive, breakbeats — finger extension controls BPM (60-200) |
| **Fist** | Cycles through preset mood labels |
| **Spacebar** | Panic stop (1 second mute) |

### Groove Lock

When hands stay still, the music **stays in its groove** — no prompt changes are sent to Lyria. Small movements trigger gradual evolution ("keep the groove, shift subtly"). Large, dramatic movements trigger a direction change ("shift genre entirely").

### Transition System

The Qwen model maintains conversation history (last 6 exchanges) to ensure smooth transitions. It builds on previous themes rather than jumping randomly between styles.

---

## Architecture

### Frontend (Browser)

| File | Purpose |
|------|---------|
| `index.html` | Entry point, UI controls, import map for Three.js |
| `main.js` | Bootstraps the Game instance |
| `game.js` | MediaPipe hand tracking, gesture feature extraction, hand rendering, visualization orchestration. Tone.js replaced with a minimal shim. |
| `MusicManager.js` | **Lyria drop-in replacement** for original Tone.js synth engine. Sends gesture data to backend via WebSocket, receives PCM audio, plays via Web Audio API. Maintains full interface compatibility with game.js. |
| `DrumManager.js` | Stub module (Lyria generates all music including drums). Exports Map-based interface for game.js compatibility. |
| `WaveformVisualizer.js` | Poincare disk hyperbolic geometry shader, audio-reactive |
| `MandalaVisualizer.js` | Sacred geometry rendering with pre-allocated pools |
| `ShapeManager.js` | Pinch shape tracking between hands (white quadrilateral) |
| `ShapeTessellationShader.js` | GLSL hyperbolic tiling for shape fill |
| `DisplacementFilter.js` | SVG turbulence + smoke DOM-based warping |
| `styles.css` | Dark kiosk-style UI |

### Backend (Python)

| File | Purpose |
|------|---------|
| `backend/server.py` | FastAPI + WebSocket server. Bridges frontend gestures to Lyria + Qwen. Serves frontend static files. Implements gesture dead-zone logic (only updates on significant movement) and distinguishes gradual vs dramatic changes. |
| `backend/lyria_manager.py` | Manages Lyria RealTime WebSocket session via `google-genai` SDK. Handles connect, play/pause/stop, prompt updates, config updates (brightness, density, BPM, mute toggles), context resets, and audio chunk streaming. |
| `backend/gesture_interpreter.py` | Translates gesture data into Lyria weighted prompts via Qwen 3.5 0.8B (Ollama). Maintains conversation history for smooth transitions. Falls back to rule-based prompt generation when Ollama is unavailable. |
| `backend/requirements.txt` | Python dependencies |

### Key Technologies

| Technology | Role | Version/Model |
|-----------|------|---------------|
| **Google Lyria RealTime** | AI music generation (streaming) | `models/lyria-realtime-exp` via Gemini API v1alpha |
| **Qwen 3.5** | Gesture-to-prompt interpretation | `qwen3.5:0.8b` via Ollama (1GB, ~1s inference) |
| **MediaPipe** | Hand landmark detection | HandLandmarker v0.10.14 |
| **Three.js** | WebGL visualization | v0.161.0 |
| **FastAPI** | Backend WebSocket server | Latest |
| **Web Audio API** | PCM audio playback | Browser native |

---

## Setup

### Prerequisites

- **Python 3.10+**
- **Ollama** running locally with `qwen3.5:0.8b` model
- **Google Gemini API key** with Lyria RealTime access
- **Modern browser** with webcam access (Chrome recommended)

### Installation

```bash
# Clone the repository
git clone https://github.com/SolshineCode/hyperspace-jam-lyria.git
cd hyperspace-jam-lyria

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
source .venv/Scripts/activate    # Windows/Git Bash

# Install Python dependencies
pip install -r backend/requirements.txt

# Pull the Qwen model (if not already installed)
ollama pull qwen3.5:0.8b
```

### Configuration

**Local development**: Create a `.env` file in the project root:

```
GEMINI_API_KEY=your-gemini-api-key-here
```

**Hugging Face Spaces**: Add `GEMINI_API_KEY` as a Secret in the Space settings (Settings > Repository Secrets).

The `.env` file is in `.gitignore` and will not be committed.

### Running

```bash
# Make sure Ollama is running
ollama serve  # (if not already running)

# Start the server
python -u backend/server.py

# Open in browser
# http://localhost:8088
```

The server starts on port **8088** by default. It serves both the API (WebSocket at `/ws`) and the frontend static files.

### Quick Start (one command)

```bash
bash start.sh
```

---

## Lyria RealTime API Details

### Connection

- **Protocol**: WebSocket (persistent, bidirectional, low-latency)
- **Model**: `models/lyria-realtime-exp`
- **SDK**: `google-genai` with `api_version='v1alpha'`
- **Audio format**: Raw 16-bit PCM, 48kHz, stereo
- **Chunk size**: 384,000 bytes (2 seconds of audio)
- **First chunk latency**: ~3-4 seconds after `play()`
- **Session limit**: 10 minutes (auto-reconnect planned)

### Controls Used

| Parameter | Range | Mapped From |
|-----------|-------|-------------|
| `weighted_prompts` | Text + weight (0.1-3.0) | Qwen 3.5 output from gesture interpretation |
| `brightness` | 0.0-1.0 | Hand height * wrist tilt factor |
| `density` | 0.0-1.0 | Finger spread * volume |
| `bpm` | 60-200 | Drum hand finger extension average (requires context reset) |
| `mute_drums` | bool | Gesture flag |
| `mute_bass` | bool | Gesture flag |
| `temperature` | 0.0-3.0 | Fixed at 1.1 (default) |
| `guidance` | 0.0-6.0 | Fixed at 4.0 (prompt adherence) |

### Prompt Engineering

The Qwen model receives structured gesture descriptions like:

```
Hand HIGH (0.8), fingers WIDE, wrist STRAIGHT, movement STILL.

Evolve gradually — keep the groove, shift subtly.
```

And outputs weighted prompt JSON:

```json
{
  "prompts": [
    {"text": "ethereal ambient floating pads celestial", "weight": 1.5},
    {"text": "shimmering reverb atmosphere", "weight": 1.0}
  ]
}
```

The conversation history ensures coherent transitions between styles.

---

## Gesture Change Detection

The backend implements a two-tier change detection system:

### Significant Change (triggers config update + Qwen prompt)
- Hand height moved > 0.12
- Finger spread changed > 0.15
- Wrist angle changed > 0.25
- Hand velocity > 0.25
- Shape (white square) toggled
- Drum hand toggled

### Major Change (triggers Qwen "change direction" mode)
- Hand height moved > 0.3
- Finger spread changed > 0.35
- Hand velocity > 0.5
- Shape toggled

If neither threshold is met, **no updates are sent** — the music stays locked in its current groove.

---

## Deployment to Hugging Face Spaces

1. Create a **private** Space on Hugging Face (SDK: Docker or Static)
2. Add `GEMINI_API_KEY` as a Secret in Space settings
3. Push the repository to the Space
4. The backend will read the API key from environment variables

---

## Known Limitations

- **Lyria session limit**: 10 minutes max per session. Page refresh starts a new session.
- **Audio latency**: ~3-4 second initial latency for first chunk, then ~2 second chunks arrive smoothly. Not suitable for rhythm-game-level precision.
- **Qwen inference**: ~1-2 seconds per prompt generation on CPU with 0.8B model. Doesn't block audio streaming.
- **No vocals**: Lyria RealTime generates instrumental music only.
- **Experimental API**: Lyria RealTime is in `v1alpha` — API may change.

---

## Credits

- **Original Hyperspace Jam**: [collidingScopes](https://github.com/collidingScopes/arpeggiator) — hand tracking + Tone.js synth + hyperbolic visuals
- **Lyria RealTime**: [Google DeepMind](https://deepmind.google/models/lyria/lyria-realtime/) — real-time AI music generation
- **Qwen 3.5**: [Alibaba/Qwen](https://github.com/QwenLM/Qwen3) — small language model for gesture interpretation
- **MediaPipe**: [Google](https://mediapipe.dev/) — hand landmark detection
- **Three.js**: [mrdoob](https://threejs.org/) — WebGL rendering
- **Ollama**: [ollama.com](https://ollama.com/) — local LLM inference

## License

MIT License