Update README.md
Browse files
README.md
CHANGED
|
@@ -5,7 +5,7 @@ base_model: Qwen/Qwen3-ASR-0.6B-hf
|
|
| 5 |
pipeline_tag: audio-classification
|
| 6 |
tags: [turn-detection, end-of-turn, turn-taking, voice-agent, endpointing, semantic-vad, speech]
|
| 7 |
model-index:
|
| 8 |
-
- name: voxturn
|
| 9 |
results:
|
| 10 |
- task: {type: audio-classification, name: End-of-turn detection}
|
| 11 |
dataset: {type: livekit/eot-bench-data, name: eot-bench (English, validation)}
|
|
@@ -16,19 +16,28 @@ model-index:
|
|
| 16 |
- {type: latency_at_10pct_cutoff, value: 335, name: Latency @ 10% cut-offs (ms)}
|
| 17 |
---
|
| 18 |
|
| 19 |
-
# voxturn
|
| 20 |
|
| 21 |
-
**An open audio turn-taking model that answers several questions about the moment in one forward pass**: has the
|
| 22 |
user finished their turn, how soon will they go on, is this a backchannel or an interruption, is the sound speech at
|
| 23 |
all. It hears the last 8 seconds of user audio (optionally with a transcript or the dialog so far), runs on one GPU in
|
| 24 |
-
~8 ms, and is measured on all 14 languages of [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
| | |
|
| 27 |
|---|---|
|
| 28 |
| **English** | 11.5% false cut-offs at a 300 ms latency budget — ahead of the commercial Baton (12.3%) and Deepgram Flux (12.9%) |
|
| 29 |
| **Multilingual** | 19.9% mean over 14 languages — the best published mean among open models (LiveKit v1-mini 28.5%, Smart Turn v3.2 39.8%) |
|
| 30 |
| **One pass, many questions** | 4 questions in 9.0 ms vs 33.2 ms asked one by one (3.7×), answers identical up to rounding |
|
| 31 |
-
| **Size** | one 0.8B model; 8.3 ms per decision, 1.9 GB GPU memory |
|
| 32 |
| **Drop-in for voice agents** | turn analyzers for [Pipecat](#use-with-pipecat) (16.7% of turns cut off vs 45.3% with Smart Turn v3) and [LiveKit Agents](#use-with-livekit-agents) (9.3% vs 18.0% with LiveKit's local turn detector, at lower latency) |
|
| 33 |
|
| 34 |
## Quick start
|
|
@@ -131,6 +140,8 @@ recheck_secs=...)`). Create the detector once per worker process (e.g. in `prewa
|
|
| 131 |
|
| 132 |
## What it answers
|
| 133 |
|
|
|
|
|
|
|
| 134 |
Every question is a multiple-choice prompt; all of them share one prefix (audio, optional text, who is speaking) and
|
| 135 |
are answered in the same forward pass.
|
| 136 |
|
|
@@ -152,12 +163,13 @@ and Chinese (`lang`); the audio itself can be in any language.
|
|
| 152 |
| Backbone | [Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0) |
|
| 153 |
| Readout | no text is generated: each question is written out with its options, and a linear head scores the hidden state at each option's last token; softmax over the options |
|
| 154 |
| One pass | question blocks follow a shared prefix under a block-isolating attention mask, each at the positions it would have if asked alone — so one pass with k questions equals k separate passes |
|
| 155 |
-
| Files | `voxturn-v1.safetensors` (47 MB), `voxturn.py` (inference), `pipecat_voxturn.py` (Pipecat analyzer), `livekit_voxturn.py` (LiveKit turn detector), `questions.json` (prompts), `config.json` |
|
| 156 |
|
| 157 |
## Evaluation
|
| 158 |
|
| 159 |
### eot-bench (validation, 100 ms minimum silence)
|
| 160 |
|
|
|
|
|
|
|
| 161 |
Metrics from `eot-harness`: false cut-offs at a latency budget (lower is better) and latency at a cut-off rate. eot-bench
|
| 162 |
audio and labels were never used for training.
|
| 163 |
|
|
@@ -198,6 +210,8 @@ the 14 languages and has the best mean.
|
|
| 198 |
|
| 199 |
### Speed (NVIDIA L40S, bf16, 8 s audio)
|
| 200 |
|
|
|
|
|
|
|
| 201 |
| | p50 | p95 |
|
| 202 |
|---|---|---|
|
| 203 |
| 1 question (`eot`) | 8.3 ms | 8.4 ms |
|
|
@@ -205,10 +219,10 @@ the 14 languages and has the best mean.
|
|
| 205 |
| the same 4 questions as 4 passes | 33.2 ms | 33.3 ms |
|
| 206 |
| 4 clips in one batch, 1 question each | 16.0 ms | 16.1 ms |
|
| 207 |
|
| 208 |
-
End
|
| 209 |
adapter weights into the backbone and replays CUDA graphs captured on first use of each input shape (the very first
|
| 210 |
-
call takes
|
| 211 |
-
(
|
| 212 |
|
| 213 |
Training details will be described in a technical report.
|
| 214 |
|
|
|
|
| 5 |
pipeline_tag: audio-classification
|
| 6 |
tags: [turn-detection, end-of-turn, turn-taking, voice-agent, endpointing, semantic-vad, speech]
|
| 7 |
model-index:
|
| 8 |
+
- name: voxturn
|
| 9 |
results:
|
| 10 |
- task: {type: audio-classification, name: End-of-turn detection}
|
| 11 |
dataset: {type: livekit/eot-bench-data, name: eot-bench (English, validation)}
|
|
|
|
| 16 |
- {type: latency_at_10pct_cutoff, value: 335, name: Latency @ 10% cut-offs (ms)}
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# voxturn
|
| 20 |
|
| 21 |
+
<!-- **An open audio turn-taking model that answers several questions about the moment in one forward pass**: has the
|
| 22 |
user finished their turn, how soon will they go on, is this a backchannel or an interruption, is the sound speech at
|
| 23 |
all. It hears the last 8 seconds of user audio (optionally with a transcript or the dialog so far), runs on one GPU in
|
| 24 |
+
~8 ms, and is measured on all 14 languages of [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data). -->
|
| 25 |
+
|
| 26 |
+
**An open, audio-conditioned turn-taking model that answers several conversational questions in one forward pass.**
|
| 27 |
+
Has the user finished their turn? When will they resume? Is this feedback or an interruption? Is this a non-speech
|
| 28 |
+
sound? VoxTurn analyzes the last 8 seconds of audio, optionally with a transcript or dialog context. On an NVIDIA
|
| 29 |
+
L40S (BF16, optimized inference), one end-of-turn decision takes **8.3 ms p50**, including feature extraction.
|
| 30 |
+
End-of-turn detection is evaluated in 14 languages on [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data).
|
| 31 |
+
|
| 32 |
+

|
| 33 |
+
|
| 34 |
|
| 35 |
| | |
|
| 36 |
|---|---|
|
| 37 |
| **English** | 11.5% false cut-offs at a 300 ms latency budget — ahead of the commercial Baton (12.3%) and Deepgram Flux (12.9%) |
|
| 38 |
| **Multilingual** | 19.9% mean over 14 languages — the best published mean among open models (LiveKit v1-mini 28.5%, Smart Turn v3.2 39.8%) |
|
| 39 |
| **One pass, many questions** | 4 questions in 9.0 ms vs 33.2 ms asked one by one (3.7×), answers identical up to rounding |
|
| 40 |
+
| **Size** | one 0.8B model; 8.3 ms per decision on Nvidia L40S, 1.9 GB GPU memory |
|
| 41 |
| **Drop-in for voice agents** | turn analyzers for [Pipecat](#use-with-pipecat) (16.7% of turns cut off vs 45.3% with Smart Turn v3) and [LiveKit Agents](#use-with-livekit-agents) (9.3% vs 18.0% with LiveKit's local turn detector, at lower latency) |
|
| 42 |
|
| 43 |
## Quick start
|
|
|
|
| 140 |
|
| 141 |
## What it answers
|
| 142 |
|
| 143 |
+

|
| 144 |
+
|
| 145 |
Every question is a multiple-choice prompt; all of them share one prefix (audio, optional text, who is speaking) and
|
| 146 |
are answered in the same forward pass.
|
| 147 |
|
|
|
|
| 163 |
| Backbone | [Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0) |
|
| 164 |
| Readout | no text is generated: each question is written out with its options, and a linear head scores the hidden state at each option's last token; softmax over the options |
|
| 165 |
| One pass | question blocks follow a shared prefix under a block-isolating attention mask, each at the positions it would have if asked alone — so one pass with k questions equals k separate passes |
|
|
|
|
| 166 |
|
| 167 |
## Evaluation
|
| 168 |
|
| 169 |
### eot-bench (validation, 100 ms minimum silence)
|
| 170 |
|
| 171 |
+

|
| 172 |
+
|
| 173 |
Metrics from `eot-harness`: false cut-offs at a latency budget (lower is better) and latency at a cut-off rate. eot-bench
|
| 174 |
audio and labels were never used for training.
|
| 175 |
|
|
|
|
| 210 |
|
| 211 |
### Speed (NVIDIA L40S, bf16, 8 s audio)
|
| 212 |
|
| 213 |
+

|
| 214 |
+
|
| 215 |
| | p50 | p95 |
|
| 216 |
|---|---|---|
|
| 217 |
| 1 question (`eot`) | 8.3 ms | 8.4 ms |
|
|
|
|
| 219 |
| the same 4 questions as 4 passes | 33.2 ms | 33.3 ms |
|
| 220 |
| 4 clips in one batch, 1 question each | 16.0 ms | 16.1 ms |
|
| 221 |
|
| 222 |
+
End-to-end inference time through `predict`, including feature extraction (not end-of-turn response latency); GPU memory 1.9 GB. The default `fast=True` merges the
|
| 223 |
adapter weights into the backbone and replays CUDA graphs captured on first use of each input shape (the very first
|
| 224 |
+
call takes approximately 0.5 s, each new shape approximately 0.1 s). `VoxTurn.from_pretrained(path, fast=False)` runs the unmerged weights eagerly
|
| 225 |
+
(approximately 38 ms) and reproduces the evaluation computation exactly.
|
| 226 |
|
| 227 |
Training details will be described in a technical report.
|
| 228 |
|