Phonon 31m English ONNX
ONNX export of gradium/phonon-31m-en (revision 541f5752bc3c79583068bb205bfda5eb39aaa08a), the 31M-parameter English text-to-speech model by Gradium, built on Kyutai's Pocket TTS and the Mimi codec, with the 37 English voices of gradium/phonon-voices (revision 0ee17205f254b5abc43b266c8a0f215565c352f7). It runs in plain onnxruntime and through the phonon engine of slicc-ortllama (POST /v1/audio/speech).
All credit for the model and the voices goes to Gradium. This repo only converts them.
License
Apache-2.0, like the model (gradium/phonon-31m-en) and the voices (gradium/phonon-voices) it is made from, both published by Gradium under Apache-2.0.
Files
| File | What |
|---|---|
config.json |
the checkpoint's own config, with an onnx section: graph shapes, the Mimi decoder's state tensors, the voices, the default voice (easton) and the graphs' extra inputs with their defaults (duration_delta: 0) |
tokenizer.json |
the checkpoint's tokenizer (Unigram with byte fallback) |
onnx/text_conditioner.onnx |
token IDs to text embeddings, plus the first step's embedding for a duration_delta |
onnx/flow_lm_main.onnx |
the flow LM transformer: embeddings and past_key_values.* to the last hidden state, the end-of-speech logit and present.* |
onnx/flow_lm_flow.onnx |
the flow head: hidden state and noise to one 32-dim Mimi latent, and the next step's embedding for a duration_delta |
onnx/mimi_decoder.onnx |
the streaming Mimi decoder: N latents, an offset and 13 state tensors to 1920ยทN samples at 24 kHz and the next state |
voices/<name>.bin |
voice prompts, already projected for the flow LM: [125, 512] float32, little-endian |
export/ |
the recipe that made this export, with its environment and provenance |
All graphs are opset 17. Variants:
| Variant | Suffix | What | Size |
|---|---|---|---|
fp32 |
none | float32 | 164 MB |
q8 |
_quantized |
8-bit MatMulNBits (block 32, symmetric, accuracy level 4) on the flow LM; Mimi in fp32 |
82 MB |
q4 |
_q4 |
4-bit MatMulNBits (block 32, symmetric, accuracy level 4) on the flow LM; Mimi in fp32 |
68 MB |
fp16 |
_fp16 |
float16 weights and activations, float32 inputs and outputs; RoPE's Sin, Cos and Range in fp32 | 82 MB |
A graph without a variant file (the text conditioner and Mimi decoder of q8 and q4) is used in fp32.
Voices (37): zoey, juniper, everly, harper, hadley, delia, easton, micah, trevor, russell, preston, miles, maeve, freya, pippa, elodie-rose, imogen, lottie, alfie, hugh, edwin, spencer, toby, teddy, bryony, captain-silas-blackwater, morwenna-of-the-deep, old-mortlake, azgorath, fennick-the-young-elf, the-thing-under-the-stairs, vyrmathrax-the-ancient, grub-the-ogre, master-aldric, the-snow-queen, sir-bramwell, fairy-godmother.
Parity
Checked against Gradium's runtime, ptts 0.4.1 (Rust, gradium-ai/xn-ptts at 49b9033e44ab0fbb935973eca3213757644d1bbb), in f32 on the CPU: 20 English sentences in the voice Easton, temperature 0.3, seed 42, 949 frames, with ptts's noise replayed into onnxruntime.
Teacher-forced (each frame starts from ptts's previous latent; the Mimi decoder decodes ptts's latents):
| Variant | Latent max abs error | Relative error, median / p99 | End-of-speech logit max error | Mimi audio SNR, min / median |
|---|---|---|---|---|
fp32 |
3.9e-05 | 7.1e-07 / 2.4e-06 | 1.1e-05 | 115.8 / 118.7 dB |
q8 |
4.2e+00 | 1.6e-02 / 8.5e-02 | 1.9e-01 | 115.8 / 118.7 dB |
q4 |
2.8e+00 | 1.5e-01 / 4.3e-01 | 1.2e+00 | 115.8 / 118.7 dB |
fp16 |
3.6e-02 | 3.7e-04 / 2.1e-03 | 7.7e-03 | 58.0 / 60.2 dB |
Free-running fp32 with ptts's noise: stops on the same frame as ptts in 20 of 20 sentences, PCM SNR 33.6 dB min and 63.8 dB median (f32 rounding compounds over the autoregression).
With duration_delta at -0.3 and 0.3 (ptts's --condition duration_delta=โฆ), teacher-forced the same way:
duration_delta |
Variant | Latent max abs error | Relative error, median / p99 | End-of-speech logit max error | Mimi audio SNR, min / median |
|---|---|---|---|---|---|
| -0.3 | fp32 |
5.4e-05 | 7.5e-07 / 3.4e-06 | 1.7e-05 | 116.6 / 119.4 dB |
| -0.3 | q8 |
1.0e+00 | 1.6e-02 / 8.2e-02 | 1.8e-01 | 116.6 / 119.4 dB |
| -0.3 | q4 |
3.1e+00 | 1.5e-01 / 4.8e-01 | 1.4e+00 | 116.6 / 119.4 dB |
| -0.3 | fp16 |
2.9e-02 | 4.0e-04 / 1.8e-03 | 8.0e-03 | 57.3 / 60.6 dB |
| 0.3 | fp32 |
4.2e-05 | 6.9e-07 / 4.6e-06 | 1.5e-05 | 118.0 / 118.7 dB |
| 0.3 | q8 |
1.7e+00 | 1.5e-02 / 9.5e-02 | 2.3e-01 | 118.0 / 118.7 dB |
| 0.3 | q4 |
3.6e+00 | 1.5e-01 / 4.8e-01 | 1.7e+00 | 118.0 / 118.7 dB |
| 0.3 | fp16 |
2.6e-02 | 3.8e-04 / 2.6e-03 | 7.9e-03 | 58.0 / 60.5 dB |
Free-running fp32: at -0.3, stops on the same frame as ptts in 20 of 20 sentences, PCM SNR 37.4 dB min and 68.8 dB median; at 0.3, stops on the same frame as ptts in 19 of 20 sentences, PCM SNR 25.9 dB min and 61.1 dB median, not counting 1 that stops on another frame on an end-of-speech near-tie (ptts's logit within 0.1 of the threshold) and 1 whose ptts trajectory a float64 PyTorch run with the same noise misses too.
Voices: every voices/<name>.bin matches ptts's projection of the voice within 2e-06.
Text: slicc-ortllama 1.25.0's English normalizer and tokenizer give ptts's normalized text, chunks and token IDs on 260 of 260 test inputs and 3000 of 3000 fuzzed strings.
Use
With slicc-ortllama 1.25.0 or later, which reads models from the Hugging Face cache:
npm install -g @ai-ecoverse/slicc-ortllama onnxruntime-node
hf download ai-ecoverse/phonon-31m-en-ONNX
ortllama serve
curl -s http://127.0.0.1:11435/v1/audio/speech -H 'content-type: application/json' \
-d '{"model": "ai-ecoverse/phonon-31m-en-ONNX", "input": "Hello from Phonon.", "voice": "easton"}' -o hello.mp3
In plain onnxruntime: feed the voice prompt through flow_lm_main from an empty cache, then the text embeddings of each chunk, then one frame at a time: flow_lm_main on the previous step's embedding gives the hidden state and the end-of-speech logit, flow_lm_flow turns it and Gaussian noise (variance = temperature) into a latent and the next embedding, and mimi_decoder turns latents into audio. text_conditioner and flow_lm_flow also take duration_delta, a float32 [1] for the model's duration conditioner: 0 is its default pace, and every step of an utterance needs the same value. Stop a few frames after the end-of-speech logit passes eos_threshold.
- Downloads last month
- 56
Model tree for ai-ecoverse/phonon-31m-en-ONNX
Base model
gradium/phonon-31m-en