Phonon 31m English ONNX

ONNX export of gradium/phonon-31m-en (revision 541f5752bc3c79583068bb205bfda5eb39aaa08a), the 31M-parameter English text-to-speech model by Gradium, built on Kyutai's Pocket TTS and the Mimi codec, with the 37 English voices of gradium/phonon-voices (revision 0ee17205f254b5abc43b266c8a0f215565c352f7). It runs in plain onnxruntime and through the phonon engine of slicc-ortllama (POST /v1/audio/speech).

All credit for the model and the voices goes to Gradium. This repo only converts them.

License

Apache-2.0, like the model (gradium/phonon-31m-en) and the voices (gradium/phonon-voices) it is made from, both published by Gradium under Apache-2.0.

Files

File What
config.json the checkpoint's own config, with an onnx section: graph shapes, the Mimi decoder's state tensors, the voices, the default voice (easton) and the graphs' extra inputs with their defaults (duration_delta: 0)
tokenizer.json the checkpoint's tokenizer (Unigram with byte fallback)
onnx/text_conditioner.onnx token IDs to text embeddings, plus the first step's embedding for a duration_delta
onnx/flow_lm_main.onnx the flow LM transformer: embeddings and past_key_values.* to the last hidden state, the end-of-speech logit and present.*
onnx/flow_lm_flow.onnx the flow head: hidden state and noise to one 32-dim Mimi latent, and the next step's embedding for a duration_delta
onnx/mimi_decoder.onnx the streaming Mimi decoder: N latents, an offset and 13 state tensors to 1920ยทN samples at 24 kHz and the next state
voices/<name>.bin voice prompts, already projected for the flow LM: [125, 512] float32, little-endian
export/ the recipe that made this export, with its environment and provenance

All graphs are opset 17. Variants:

Variant Suffix What Size
fp32 none float32 164 MB
q8 _quantized 8-bit MatMulNBits (block 32, symmetric, accuracy level 4) on the flow LM; Mimi in fp32 82 MB
q4 _q4 4-bit MatMulNBits (block 32, symmetric, accuracy level 4) on the flow LM; Mimi in fp32 68 MB
fp16 _fp16 float16 weights and activations, float32 inputs and outputs; RoPE's Sin, Cos and Range in fp32 82 MB

A graph without a variant file (the text conditioner and Mimi decoder of q8 and q4) is used in fp32.

Voices (37): zoey, juniper, everly, harper, hadley, delia, easton, micah, trevor, russell, preston, miles, maeve, freya, pippa, elodie-rose, imogen, lottie, alfie, hugh, edwin, spencer, toby, teddy, bryony, captain-silas-blackwater, morwenna-of-the-deep, old-mortlake, azgorath, fennick-the-young-elf, the-thing-under-the-stairs, vyrmathrax-the-ancient, grub-the-ogre, master-aldric, the-snow-queen, sir-bramwell, fairy-godmother.

Parity

Checked against Gradium's runtime, ptts 0.4.1 (Rust, gradium-ai/xn-ptts at 49b9033e44ab0fbb935973eca3213757644d1bbb), in f32 on the CPU: 20 English sentences in the voice Easton, temperature 0.3, seed 42, 949 frames, with ptts's noise replayed into onnxruntime.

Teacher-forced (each frame starts from ptts's previous latent; the Mimi decoder decodes ptts's latents):

Variant Latent max abs error Relative error, median / p99 End-of-speech logit max error Mimi audio SNR, min / median
fp32 3.9e-05 7.1e-07 / 2.4e-06 1.1e-05 115.8 / 118.7 dB
q8 4.2e+00 1.6e-02 / 8.5e-02 1.9e-01 115.8 / 118.7 dB
q4 2.8e+00 1.5e-01 / 4.3e-01 1.2e+00 115.8 / 118.7 dB
fp16 3.6e-02 3.7e-04 / 2.1e-03 7.7e-03 58.0 / 60.2 dB

Free-running fp32 with ptts's noise: stops on the same frame as ptts in 20 of 20 sentences, PCM SNR 33.6 dB min and 63.8 dB median (f32 rounding compounds over the autoregression).

With duration_delta at -0.3 and 0.3 (ptts's --condition duration_delta=โ€ฆ), teacher-forced the same way:

duration_delta Variant Latent max abs error Relative error, median / p99 End-of-speech logit max error Mimi audio SNR, min / median
-0.3 fp32 5.4e-05 7.5e-07 / 3.4e-06 1.7e-05 116.6 / 119.4 dB
-0.3 q8 1.0e+00 1.6e-02 / 8.2e-02 1.8e-01 116.6 / 119.4 dB
-0.3 q4 3.1e+00 1.5e-01 / 4.8e-01 1.4e+00 116.6 / 119.4 dB
-0.3 fp16 2.9e-02 4.0e-04 / 1.8e-03 8.0e-03 57.3 / 60.6 dB
0.3 fp32 4.2e-05 6.9e-07 / 4.6e-06 1.5e-05 118.0 / 118.7 dB
0.3 q8 1.7e+00 1.5e-02 / 9.5e-02 2.3e-01 118.0 / 118.7 dB
0.3 q4 3.6e+00 1.5e-01 / 4.8e-01 1.7e+00 118.0 / 118.7 dB
0.3 fp16 2.6e-02 3.8e-04 / 2.6e-03 7.9e-03 58.0 / 60.5 dB

Free-running fp32: at -0.3, stops on the same frame as ptts in 20 of 20 sentences, PCM SNR 37.4 dB min and 68.8 dB median; at 0.3, stops on the same frame as ptts in 19 of 20 sentences, PCM SNR 25.9 dB min and 61.1 dB median, not counting 1 that stops on another frame on an end-of-speech near-tie (ptts's logit within 0.1 of the threshold) and 1 whose ptts trajectory a float64 PyTorch run with the same noise misses too.

Voices: every voices/<name>.bin matches ptts's projection of the voice within 2e-06.

Text: slicc-ortllama 1.25.0's English normalizer and tokenizer give ptts's normalized text, chunks and token IDs on 260 of 260 test inputs and 3000 of 3000 fuzzed strings.

Use

With slicc-ortllama 1.25.0 or later, which reads models from the Hugging Face cache:

npm install -g @ai-ecoverse/slicc-ortllama onnxruntime-node
hf download ai-ecoverse/phonon-31m-en-ONNX
ortllama serve
curl -s http://127.0.0.1:11435/v1/audio/speech -H 'content-type: application/json' \
  -d '{"model": "ai-ecoverse/phonon-31m-en-ONNX", "input": "Hello from Phonon.", "voice": "easton"}' -o hello.mp3

In plain onnxruntime: feed the voice prompt through flow_lm_main from an empty cache, then the text embeddings of each chunk, then one frame at a time: flow_lm_main on the previous step's embedding gives the hidden state and the end-of-speech logit, flow_lm_flow turns it and Gaussian noise (variance = temperature) into a latent and the next embedding, and mimi_decoder turns latents into audio. text_conditioner and flow_lm_flow also take duration_delta, a float32 [1] for the model's duration conditioner: 0 is its default pace, and every step of an utterance needs the same value. Stop a few frames after the end-of-speech logit passes eos_threshold.

Downloads last month
56
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ai-ecoverse/phonon-31m-en-ONNX

Quantized
(17)
this model