AustinXiao commited on
Commit
1192531
·
verified ·
1 Parent(s): 7ad6886

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +23 -9
README.md CHANGED
@@ -5,7 +5,7 @@ base_model: Qwen/Qwen3-ASR-0.6B-hf
5
  pipeline_tag: audio-classification
6
  tags: [turn-detection, end-of-turn, turn-taking, voice-agent, endpointing, semantic-vad, speech]
7
  model-index:
8
- - name: voxturn-v1
9
  results:
10
  - task: {type: audio-classification, name: End-of-turn detection}
11
  dataset: {type: livekit/eot-bench-data, name: eot-bench (English, validation)}
@@ -16,19 +16,28 @@ model-index:
16
  - {type: latency_at_10pct_cutoff, value: 335, name: Latency @ 10% cut-offs (ms)}
17
  ---
18
 
19
- # voxturn v1
20
 
21
- **An open audio turn-taking model that answers several questions about the moment in one forward pass**: has the
22
  user finished their turn, how soon will they go on, is this a backchannel or an interruption, is the sound speech at
23
  all. It hears the last 8 seconds of user audio (optionally with a transcript or the dialog so far), runs on one GPU in
24
- ~8 ms, and is measured on all 14 languages of [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data).
 
 
 
 
 
 
 
 
 
25
 
26
  | | |
27
  |---|---|
28
  | **English** | 11.5% false cut-offs at a 300 ms latency budget — ahead of the commercial Baton (12.3%) and Deepgram Flux (12.9%) |
29
  | **Multilingual** | 19.9% mean over 14 languages — the best published mean among open models (LiveKit v1-mini 28.5%, Smart Turn v3.2 39.8%) |
30
  | **One pass, many questions** | 4 questions in 9.0 ms vs 33.2 ms asked one by one (3.7×), answers identical up to rounding |
31
- | **Size** | one 0.8B model; 8.3 ms per decision, 1.9 GB GPU memory |
32
  | **Drop-in for voice agents** | turn analyzers for [Pipecat](#use-with-pipecat) (16.7% of turns cut off vs 45.3% with Smart Turn v3) and [LiveKit Agents](#use-with-livekit-agents) (9.3% vs 18.0% with LiveKit's local turn detector, at lower latency) |
33
 
34
  ## Quick start
@@ -131,6 +140,8 @@ recheck_secs=...)`). Create the detector once per worker process (e.g. in `prewa
131
 
132
  ## What it answers
133
 
 
 
134
  Every question is a multiple-choice prompt; all of them share one prefix (audio, optional text, who is speaking) and
135
  are answered in the same forward pass.
136
 
@@ -152,12 +163,13 @@ and Chinese (`lang`); the audio itself can be in any language.
152
  | Backbone | [Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0) |
153
  | Readout | no text is generated: each question is written out with its options, and a linear head scores the hidden state at each option's last token; softmax over the options |
154
  | One pass | question blocks follow a shared prefix under a block-isolating attention mask, each at the positions it would have if asked alone — so one pass with k questions equals k separate passes |
155
- | Files | `voxturn-v1.safetensors` (47 MB), `voxturn.py` (inference), `pipecat_voxturn.py` (Pipecat analyzer), `livekit_voxturn.py` (LiveKit turn detector), `questions.json` (prompts), `config.json` |
156
 
157
  ## Evaluation
158
 
159
  ### eot-bench (validation, 100 ms minimum silence)
160
 
 
 
161
  Metrics from `eot-harness`: false cut-offs at a latency budget (lower is better) and latency at a cut-off rate. eot-bench
162
  audio and labels were never used for training.
163
 
@@ -198,6 +210,8 @@ the 14 languages and has the best mean.
198
 
199
  ### Speed (NVIDIA L40S, bf16, 8 s audio)
200
 
 
 
201
  | | p50 | p95 |
202
  |---|---|---|
203
  | 1 question (`eot`) | 8.3 ms | 8.4 ms |
@@ -205,10 +219,10 @@ the 14 languages and has the best mean.
205
  | the same 4 questions as 4 passes | 33.2 ms | 33.3 ms |
206
  | 4 clips in one batch, 1 question each | 16.0 ms | 16.1 ms |
207
 
208
- End to end through `predict`, including feature extraction; GPU memory 1.9 GB. The default `fast=True` merges the
209
  adapter weights into the backbone and replays CUDA graphs captured on first use of each input shape (the very first
210
- call takes ~0.5 s, each new shape ~0.1 s). `VoxTurn.from_pretrained(path, fast=False)` runs the unmerged weights eagerly
211
- (~38 ms) and reproduces the evaluation computation exactly.
212
 
213
  Training details will be described in a technical report.
214
 
 
5
  pipeline_tag: audio-classification
6
  tags: [turn-detection, end-of-turn, turn-taking, voice-agent, endpointing, semantic-vad, speech]
7
  model-index:
8
+ - name: voxturn
9
  results:
10
  - task: {type: audio-classification, name: End-of-turn detection}
11
  dataset: {type: livekit/eot-bench-data, name: eot-bench (English, validation)}
 
16
  - {type: latency_at_10pct_cutoff, value: 335, name: Latency @ 10% cut-offs (ms)}
17
  ---
18
 
19
+ # voxturn
20
 
21
+ <!-- **An open audio turn-taking model that answers several questions about the moment in one forward pass**: has the
22
  user finished their turn, how soon will they go on, is this a backchannel or an interruption, is the sound speech at
23
  all. It hears the last 8 seconds of user audio (optionally with a transcript or the dialog so far), runs on one GPU in
24
+ ~8 ms, and is measured on all 14 languages of [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data). -->
25
+
26
+ **An open, audio-conditioned turn-taking model that answers several conversational questions in one forward pass.**
27
+ Has the user finished their turn? When will they resume? Is this feedback or an interruption? Is this a non-speech
28
+ sound? VoxTurn analyzes the last 8 seconds of audio, optionally with a transcript or dialog context. On an NVIDIA
29
+ L40S (BF16, optimized inference), one end-of-turn decision takes **8.3 ms p50**, including feature extraction.
30
+ End-of-turn detection is evaluated in 14 languages on [eot-bench](https://huggingface.co/datasets/livekit/eot-bench-data).
31
+
32
+ ![Figure (a). Conversational timing: mid-turn pauses, completed turns, backchannels, and interruptions call for different agent behavior.](figures/a_conversational_timing.png)
33
+
34
 
35
  | | |
36
  |---|---|
37
  | **English** | 11.5% false cut-offs at a 300 ms latency budget — ahead of the commercial Baton (12.3%) and Deepgram Flux (12.9%) |
38
  | **Multilingual** | 19.9% mean over 14 languages — the best published mean among open models (LiveKit v1-mini 28.5%, Smart Turn v3.2 39.8%) |
39
  | **One pass, many questions** | 4 questions in 9.0 ms vs 33.2 ms asked one by one (3.7×), answers identical up to rounding |
40
+ | **Size** | one 0.8B model; 8.3 ms per decision on Nvidia L40S, 1.9 GB GPU memory |
41
  | **Drop-in for voice agents** | turn analyzers for [Pipecat](#use-with-pipecat) (16.7% of turns cut off vs 45.3% with Smart Turn v3) and [LiveKit Agents](#use-with-livekit-agents) (9.3% vs 18.0% with LiveKit's local turn detector, at lower latency) |
42
 
43
  ## Quick start
 
140
 
141
  ## What it answers
142
 
143
+ ![Figure (b). Shared audio prefix and block-isolated question prompting in VoxTurn v1.](figures/b_model_architecture.png)
144
+
145
  Every question is a multiple-choice prompt; all of them share one prefix (audio, optional text, who is speaking) and
146
  are answered in the same forward pass.
147
 
 
163
  | Backbone | [Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0) |
164
  | Readout | no text is generated: each question is written out with its options, and a linear head scores the hidden state at each option's last token; softmax over the options |
165
  | One pass | question blocks follow a shared prefix under a block-isolating attention mask, each at the positions it would have if asked alone — so one pass with k questions equals k separate passes |
 
166
 
167
  ## Evaluation
168
 
169
  ### eot-bench (validation, 100 ms minimum silence)
170
 
171
+ ![Figure (d). Task-specific evaluations: eot-bench, Easy-Turn, and SLR99. Lower false cut-off rates are better.](figures/d_benchmarks.png)
172
+
173
  Metrics from `eot-harness`: false cut-offs at a latency budget (lower is better) and latency at a cut-off rate. eot-bench
174
  audio and labels were never used for training.
175
 
 
210
 
211
  ### Speed (NVIDIA L40S, bf16, 8 s audio)
212
 
213
+ ![Figure (e). Single-pass inference efficiency and separate 150-turn framework replay results.](figures/e_speed_and_deployment.png)
214
+
215
  | | p50 | p95 |
216
  |---|---|---|
217
  | 1 question (`eot`) | 8.3 ms | 8.4 ms |
 
219
  | the same 4 questions as 4 passes | 33.2 ms | 33.3 ms |
220
  | 4 clips in one batch, 1 question each | 16.0 ms | 16.1 ms |
221
 
222
+ End-to-end inference time through `predict`, including feature extraction (not end-of-turn response latency); GPU memory 1.9 GB. The default `fast=True` merges the
223
  adapter weights into the backbone and replays CUDA graphs captured on first use of each input shape (the very first
224
+ call takes approximately 0.5 s, each new shape approximately 0.1 s). `VoxTurn.from_pretrained(path, fast=False)` runs the unmerged weights eagerly
225
+ (approximately 38 ms) and reproduces the evaluation computation exactly.
226
 
227
  Training details will be described in a technical report.
228