VoxLock: distilled and quantized variants

Smaller and faster versions of VoxLock (target-speaker voice isolation for calls, HF arbazsiddiqui/voxlock@2413b29) for low-end Android CPUs. Students are the same TSE-BSRNN, narrower and shallower, distilled from VoxLock on LibriSpeech calls (CC-BY-4.0) and AMI headsets (CC-BY-4.0); the Krisp benchmark was used only to evaluate. Speed-ups are against VoxLock's own chunk_int8.onnx at the same chunk size, same CPU, same native onnxruntime loop.

Use final/ in apps: chunk_*.ort (onnxruntime ORT format, graph pre-optimized for ARM, loads faster) and enroll_fp16.onnx (float16 weights, half the size). onnx/ holds every quantization as plain ONNX, torch/ the wesep-format checkpoint (config.yaml + avg_model.pt) for further training.

Students with HOP = 256 use 16 ms frames instead of 8 ms: half the work per second of audio and 16 ms algorithmic latency instead of 24. They need code/voxlock_core_hop256.patch (reads HOP from the model folder, loads .ort / fp16 enroll, and fixes the last-frame coverage in flush()); the hop-128 models load in the stock voxlock package and in sherpa-onnx.

import voxlock                                    # pip install voxlock, then apply code/voxlock_core_hop256.patch
vl = voxlock.VoxLock("x5-fd64x4-h128/final", precision="q8-mlp")
caller = vl.isolate(voxlock.load_audio("call.wav"))

Variants

Krisp Voice Isolation Benchmark, 265 recordings, Parakeet TDT 0.6B v2, the benchmark's own scoring. "vs int8" is the paired WER difference to VoxLock int8 with a caller-clustered 95% bootstrap interval. RTF: Apple M4 Pro, one thread, native onnxruntime 1.31.1 (android/voxbench.cpp), lower is faster, below 1 keeps up. Background leak = words transcribed where only the background talker speaks, per minute of such audio.

folder widthΓ—blocks hop chunk file MB (chunk + enroll) WER % vs int8 leak/min RTF 20 ms 40 ms 80 ms speed-up 20 ms
VoxLock as published, chunk_int8.onnx (baseline) 128Γ—6 128 β€” 56.6 + 21.0 5.10 β€” 0.18 0.521 0.372 0.296 Γ—1.0
teacher-q8 128Γ—6 128 final/chunk_q8_mask.ort 24.3 + 10.9 5.08 βˆ’0.02 [βˆ’0.21; +0.24] 0.15 0.398 0.317 0.269 Γ—1.3
teacher-q8 128Γ—6 128 final/chunk_q8_mlp.ort 45.3 + 10.9 5.12 +0.02 [βˆ’0.19; +0.27] 0.16 0.435 0.278 0.201 Γ—1.2
x5-fd64x4-h128 64Γ—4 128 final/chunk_q8_mask.ort 7.6 + 8.7 5.33 +0.23 [βˆ’0.28; +0.85] 1.99 0.109 0.085 0.072 Γ—4.8
x5-fd64x4-h128 64Γ—4 128 final/chunk_q8_mlp.ort 11.0 + 8.7 5.12 +0.01 [βˆ’0.41; +0.62] 1.39 0.127 0.083 0.063 Γ—4.1
x5-fd64x4-h128 64Γ—4 128 final/chunk_int8.ort 15.8 + 8.7 5.12 +0.01 [βˆ’0.35; +0.47] 1.23 0.141 0.100 0.079 Γ—3.7
x7-fd64x4-h256 64Γ—4 256 final/chunk_int8.ort 15.8 + 8.7 5.14 +0.03 [βˆ’0.35; +0.57] 1.44 0.089 0.073 0.050 Γ—5.9
x7-fd64x4-h256 64Γ—4 256 final/chunk_q8_mask.ort 7.6 + 8.7 5.24 +0.14 [βˆ’0.27; +0.68] 1.83 0.071 0.056 0.043 Γ—7.3
x7-fd64x4-h256 64Γ—4 256 final/chunk_q8_mlp.ort 11.0 + 8.7 5.37 +0.26 [βˆ’0.24; +0.94] 2.21 0.070 0.065 0.044 Γ—7.4
x7-longcall-fd64x4-h256 64Γ—4 256 final/chunk_int8.ort 15.8 + 8.7 4.94 βˆ’0.17 [βˆ’0.51; +0.23] 0.98 = x7 = x7 = x7 Γ—5.9
x7-longcall-fd64x4-h256 64Γ—4 256 final/chunk_q8_mlp.ort 11.0 + 8.7 5.03 βˆ’0.08 [βˆ’0.37; +0.34] 0.98 = x7 = x7 = x7 Γ—7.4
x11-fd48x3-h256 48Γ—3 256 final/chunk_q8_mlp.ort 6.7 + 8.2 6.48 +1.37 [+0.41; +2.59] 6.85 0.048 0.037 0.034 Γ—10.9
x11-fd48x3-h256 48Γ—3 256 final/chunk_q8_mask.ort 5.3 + 8.2 6.75 +1.64 [+0.58; +2.93] 8.11 0.048 0.035 0.025 Γ—10.9
x11-fd48x3-h256 48Γ—3 256 final/chunk_int8.ort 9.9 + 8.2 7.01 +1.91 [+0.84; +3.21] 9.16 0.050 0.041 0.028 Γ—10.4
x10-fd32x3-h128 32Γ—3 128 final/chunk_int8.ort 5.9 + 7.7 12.13 +7.02 [+4.47; +9.62] 27.69 0.054 0.037 0.027 Γ—9.6
x10-fd32x3-h128 32Γ—3 128 final/chunk_q8_mlp.ort 4.5 + 7.7 12.54 +7.44 [+4.91; +10.29] 29.57 0.054 0.036 0.026 Γ—9.6
x10-fd32x3-h128 32Γ—3 128 final/chunk_q8_mask.ort 3.9 + 7.7 12.42 +7.32 [+4.88; +10.05] 28.95 0.051 0.036 0.027 Γ—10.2
x17-fd32x2-h256 32Γ—2 256 final/chunk_q8_mlp.ort 4.2 + 7.7 8.95 +3.84 [+1.64; +6.54] 15.07 0.031 0.022 0.015 Γ—16.8
x17-fd32x2-h256 32Γ—2 256 final/chunk_int8.ort 5.8 + 7.7 9.27 +4.16 [+1.92; +6.58] 16.36 0.030 0.023 0.015 Γ—17.4
x17-fd32x2-h256 32Γ—2 256 final/chunk_q8_mask.ort 3.8 + 7.7 9.80 +4.70 [+2.25; +7.62] 18.29 0.031 0.021 0.015 Γ—16.8
x21-fd24x2-h256 24Γ—2 256 final/chunk_int8.ort 4.4 + 7.4 14.38 +9.27 [+6.15; +12.43] 35.99 0.025 0.019 0.012 Γ—20.8
x21-fd24x2-h256 24Γ—2 256 final/chunk_q8_mlp.ort 3.5 + 7.4 14.56 +9.46 [+6.47; +12.68] 36.74 0.029 0.020 0.013 Γ—18.0
x21-fd24x2-h256 24Γ—2 256 final/chunk_q8_mask.ort 3.3 + 7.4 14.28 +9.18 [+6.30; +12.31] 35.99 0.029 0.019 0.012 Γ—18.0

Which one. Start with x7-longcall-fd64x4-h256 (below): the same graph and speed as x7, Krisp WER 4.94% against x7's 5.14% with a third less background leak, and far fewer dropped caller phrases in long streams with multi-phrase voice samples. The plain x7 is x7-fd64x4-h256: VoxLock's WER (5.14% against 5.10%, paired difference +0.03, interval βˆ’0.35 to +0.57) at about Γ—6-7 the speed. For the fastest model that stays close, x11-fd48x3-h256: WER 6.5% (+1.4 points, interval +0.4 to +2.6) at about Γ—11. x5-fd64x4-h128 is the same width as x7 with 8 ms frames, for runtimes that only take hop 128 (stock voxlock, sherpa-onnx). The v1 students x10-fd32x3-h128, x17-fd32x2-h256 and x21-fd24x2-h256 lose more: WER 9-14.6% (no processing: 23.2%), so they remove roughly half to three quarters of the background speech; x11 replaces x10 at the same speed with a fifth of its WER loss.

v2 students (x7, x11) were cut from the Γ—5 student instead of from VoxLock (code/kit/distill/prune_init.py --teacher <x5 checkpoint> --hop 256) and distilled from VoxLock for 18k steps with 16 ms frames (code/kit/distill/chain_multi2.sh).

Students let more background words through than VoxLock (leak/min); listen before shipping. Peak process memory on the M4 Pro with onnxruntime: 84-90 MB for x11, 92-114 MB for x7 and 95-100 MB for x5, against 186 MB for VoxLock int8. Measurements, method and the distillation recipe: results/report.html; code: code/kit/ (training in code/kit/distill/), phone benchmark: code/phone_bench.sh.

Long calls: x7-longcall-fd64x4-h256

x7 tuned for what the app does in a call: a voice sample of up to five phrases (each phrase run through enroll on its own, outputs concatenated along time) and one endless stream whose LSTM state is never reset. Same graph, size and speed as x7 (the speed columns above are x7's; this file was not re-timed), same HOP 256 and patch.

How it was made: x7 was fine-tuned for 5,000 steps on the streaming path itself (code/longrun/longft.py: 3 s windows, LSTM state carried across windows, 1-5 phrase voice samples, LibriSpeech calls of 12-48 s cycling through caller alone / background alone / both / silence, every step also replaying x7's original distillation mix of simulated calls, real AMI headsets and two-talker clips). That model alone fixed the dropped phrases but let background speech through on real call-centre recordings (Krisp WER 5.57, not published). The published weights are 0.7 Γ— x7 + 0.3 Γ— that model (code/longrun/blend.py), which keeps most of the fix and lowers x7's background leak.

Long-call test (code/longrun/make_streams.py): five 10-minute streams, LibriSpeech dev/test voices never trained on, 199 caller utterances, state never reset, int8 chunk. A phrase is dropped when it loses more than 10 dB, weak above 3 dB.

model voice sample dropped phrases weak background suppression, median dB background blocks β‰₯ 25 dB caller loss p90, dB SI-SDRi, dB
VoxLock int8 5 phrases 12 21 38.9 80.9% 1.62 9.34
x7 5 phrases 11 22 33.6 76.6% 3.11 8.70
x7-longcall 5 phrases 1 9 37.6 78.7% 0.84 8.78
x7 1 phrase 14 27 29.7 59.6% 2.78 8.36
x7-longcall 1 phrase 7 16 32.3 70.2% 1.02 8.47

Krisp (265 recordings): WER 4.94% int8 / 5.03% q8-mlp against x7's 5.14%; paired difference to x7 βˆ’0.20 [βˆ’0.61; +0.07] (int8) and βˆ’0.11 [βˆ’0.39; +0.07] (q8-mlp); background leak 0.98/min against x7's 1.44 (VoxLock: 0.18). Only chunk_int8 and chunk_q8_mlp were evaluated, so only those are published for this model.

History and the candidates that lost (results/longrun/, Krisp WER int8 / leak per min / dropped phrases at 5 phrases): the 50/50 blend first published here (commit f5aeb2f) 5.05 / 1.38 / 1 (4 weak); a further fine-tune on long AMI headset calls with long background-only stretches (ft3, 1,000 steps) 5.52 / 3.07 / 2; its 70/30 blend with x7 5.01 / 1.21 / 3. Picked by a rule fixed before the last results came in: Krisp WER not above x7's, at most 2 dropped phrases, then the lowest leak.

Known limit: on long stretches of real speech where only the background talker speaks, the students, x7 included, let more through than VoxLock (0.98 against 0.18 words per minute on Krisp); listen before shipping.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for YamiSukkehirho/voxlock-distill

Quantized
(1)
this model