Heitorkk2/Kitsune-TTS-V1

Ultra-lightweight Text-to-Speech for anime personas and virtual assistants.

Brazilian Portuguese 🇧🇷   ·   ~39M parameters   ·   5 voices

Kitsune-TTS V1

A compact, non-autoregressive speech model built on VITS2-Slim, trained on synthetic data for expressive anime-style voices. Run it locally with PyTorch or use the ONNX export for CPU and web inference.

This repository contains the official model weights and matching configuration. The Python API, JavaScript client, training tools and examples live in the GitHub repository.

At a glance

Feature	Details
Architecture	VITS2-Slim with stochastic duration prediction and speaker embeddings
Parameters	Approximately 39M (~76 MB in FP16)
Language	Brazilian Portuguese (PT-BR)
Audio	22,050 Hz, mono
Voices	5 built-in anime character voices
Checkpoints	FP32 and compact FP16
ONNX	FP32 single-file inference graph (~115 MB), zero quantization
License	GPL-3.0

📊 Benchmarks & Performance

Measured synthesizing 9.21 seconds of audio (noise_scale=0, speaker Frieren, FP32 computation, zero quantization, identical weights):

Device / Hardware	Backend / Runtime	Checkpoint	Latency	RTF	Real-time Factor
Local CPU AMD Ryzen 7 5700U (8 threads)	PyTorch (fast_cpu) ⚡	FP16 file (~76 MB)	2.06 s	0.224	4.46×
Local CPU AMD Ryzen 7 5700U (8 threads)	PyTorch (fast_cpu) ⚡	FP32 file (~151 MB)	2.17 s	0.235	4.25×
Local CPU AMD Ryzen 7 5700U (8 threads)	ONNX Runtime CPU	FP32 ONNX (~115 MB)	2.38 s	0.259	3.86×
Local CPU AMD Ryzen 7 5700U (8 threads)	PyTorch (standard)	FP32 file (~151 MB)	3.84 s	0.417	2.40×
Cloud GPU Tesla T4 (15 GB)	PyTorch (CUDA) 🚀	FP16 file (~76 MB)	0.20 s	0.022	46.0×

💡 Highlights:

• PyTorch fast_cpu: Delivers a ~1.96× speedup over standard PyTorch CPU, outperforming ONNX Runtime while keeping 100% identical audio fidelity.
• GPU Inference: Generates ~10 seconds of speech in just 200 ms.
• Full reproduction scripts and raw data: BENCHMARK_RESULTS.md.

Meet the voices

Use the speaker key with the Python API, or the numeric ID with the JavaScript client.

ID	Voice	Speaker key	Character Origin	Style / Characteristics
0	Emilia	emilia	Re:Zero	Soft, sweet, and gentle voice
1	Frieren	frieren	Frieren	Calm, serene, and steady tone
2	Zero Two	zerotwo	Darling in the Franxx	Energetic, teasing, and playful
3	Violet	violet	Violet Evergarden	Formal, disciplined, and expressive
4	Hiro	hiro	Darling in the Franxx	Youthful, calm male voice

Choose your format

File	Use	Approximate size
latest_model_fp16.pth	Recommended compact PyTorch checkpoint (~2x lighter)	76 MB
latest_model_fp32.pth	Full-precision PyTorch checkpoint & ONNX export source	151 MB
kitsune39M.onnx	Single-file inference graph for ONNX Runtime (CPU & Web)	115 MB
model_config.json	Architecture parameters, sample rate, and speaker map	1 KB

Keep model_config.json beside the checkpoint or ONNX file. Always use weights and configuration from the same model release.

Quick start

Install eSpeak NG on your system for phonemization, then set up the repository:

git clone https://github.com/Heitorkk2/Kitsune-TTS.git
cd Kitsune-TTS
pip install -e ".[torch,onnx]" huggingface_hub scipy

PyTorch

Download the compact checkpoint and configuration:

hf download Heitorkk2/Kitsune-TTS-V1 latest_model_fp16.pth model_config.json --local-dir model

Generate your first WAV:

from scipy.io.wavfile import write
from kitsune.api import KitsuneSynthesizer

synth = KitsuneSynthesizer(
    checkpoint="model/latest_model_fp16.pth",
    device="cpu",
    fast_cpu=True,  # 🚀 ~2x faster vocoder on CPU
)
print(synth.list_speakers())

audio = synth.synthesize("Olá! Eu sou a Frieren.", speaker="frieren")
write("output.wav", synth.sample_rate, audio)

ONNX

Download the graph and configuration:

hf download Heitorkk2/Kitsune-TTS-V1 kitsune39M.onnx model_config.json --local-dir model

Run with ONNX Runtime:

from kitsune.api import KitsuneSynthesizer

synth = KitsuneSynthesizer(onnx_path="model/kitsune39M.onnx")
audio = synth.synthesize("Uma voz pequena, com muito a dizer.", speaker="emilia")

For browser/Node.js usage and export instructions, see the JavaScript client and ONNX guide.

Add your own voice

Configure your recordings and transcripts in the Colab notebook to create a separate checkpoint for one or more new speakers. The model architecture stays the same; each new speaker adds one embedding row.

Fine-tuning is experimental; defaults may evolve as we test more voices.

Usage notes

• V1 supports Brazilian Portuguese. Other languages are not supported by this release.
• Speech quality, pacing and pronunciation can vary with the input text.
• Runtime performance depends on your processor, backend, thread settings and text length.
• Use voice recordings and datasets you have permission to use.

Special thanks

Special thanks to Everteson and Nakamura for helping build the Kitsune-TTS model.

Credits & licensing

The Kitsune model weights, code and synthetic dataset are original work, licensed under GPL-3.0.

• Architecture: daniilrobnikov/vits2, MIT. Used as the architectural starting point via weight transplant from a VCTK-pretrained checkpoint. Text encoder, flow and posterior encoder layers were carried over; duration prediction, speaker embeddings and vocabulary embeddings were re-initialized from scratch to support the new model.
• Phonemization: eSpeak NG (GPL-3.0), through the phonemizer library.
• Initial transplant checkpoint: VCTK Corpus, CC BY 4.0.

Acknowledgments

• VITS2: base architecture inspiration.
• OmniVoice (k2-fsa): zero-shot voice cloning used to bootstrap the training corpus; no real recordings were used for that corpus.
• XTTS (Coqui): additional synthetic dataset generation.
• Kokoro TTS: inspiration for compact speech models.
• Piper TTS: ONNX export and CPU inference reference.

---

ᓚ₍ ^. ̫ .^₎

Made with ❤️ by Heitorkk2, Nakamura & Everteson