Qwen3-ASR-1.7B + LoRA β€” Cantonese (yue-HK) ASR (v4) - Written Language

A LoRA adapter for Qwen/Qwen3-ASR-1.7B (the official Qwen3-ASR audio-text LLM) that substantially improves Cantonese (yue-HK) speech recognition. It is fine-tuned on ~207 h of cleaned LegCo (Hong Kong Legislative Council) webcasts with official-record transcripts (written_yue) as targets. The audio tower stays frozen; LoRA (rank 16) covers the LLM self-attention and MLP projections β€” 17.4 M trainable parameters (0.85% of 2.06 B), so the adapter is a single ~70 MB file on top of the base model.

The adapter is intended for Cantonese ASR on council/debate-style speech: 12+ speakers per meeting, code-mixed English, numbers, dates, and Cantonese-informal-but-standard register. Expected output convention (official Qwen3-ASR contract): language Cantonese<asr_text> + transcript; force text-only output by prefixing the language tag.

Results

Held-out splits, WER/CER (punctuation & spaces stripped, character tokens):

Benchmark Zero-shot base This adapter relative Ξ”
v4 internal test (5,912 rows / 24 h) 0.326 0.174 βˆ’47% WER
v4 internal dev (5,891 rows / 24 h) β€” 0.163 β€”
Official-transcripts test (996 rows / 28 s avg, JackyHoCL/legco-yue-official-transcripts) 0.387 0.181 βˆ’53% WER

Checkpoint selection β€” full-split eval of every epoch (3-epoch run):

Checkpoint test WER test CER dev WER dev CER
epoch 1 (ckpt-2446) 0.178 0.179 0.172 0.171
epoch 2 (ckpt-4892) β€” released 0.174 0.173 0.163 0.163
epoch 3 (ckpt-7338) 0.176 0.176 0.165 0.164

Epoch 2 was selected by full test+dev WER (the per-epoch 48-row TensorBoard subset had suggested epoch 3).

Data

  • Source: JackyHoCL/legco-yue-asr-preview v4 snapshot (57,566 utterances / 227.9 h / 56 sessions, 16 kHz mono WAV, sliced ≀20 s)
  • Training label: written_yue (official-record written Cantonese)
  • Filters (all splits): drop rows with any QC flag and match_confidence < 0.5 β†’ 50,937 rows (train 39,134 / dev 5,891 / test 5,912, ~207 h)

Training

  • LoRA on thinker.model.layers.*.self_attn.{q,k,v,o}_proj + thinker.model.layers.*.mlp.{gate,up,down}_proj β€” rank 16, alpha 32, dropout 0.05; audio tower frozen
  • 3 epochs, lr 5e-5 cosine (warmup 5%), bf16, batch 4 Γ— grad-accum 4, gradient checkpointing, 6 h 20 m on one RTX 3080 20 GB
  • Target: language Cantonese<asr_text> + transcript (official Qwen3-ASR fine-tuning contract)

Usage β€” Transformers (this adapter)

Requires qwen-asr==0.0.6 + transformers==4.57.6 (the native transformers>=5 qwen3_asr classes do not match this checkpoint layout) and peft.

import torch
import soundfile as sf
from qwen_asr.core.transformers_backend import (
    Qwen3ASRForConditionalGeneration, Qwen3ASRProcessor)
from peft import PeftModel

processor = Qwen3ASRProcessor.from_pretrained("Qwen/Qwen3-ASR-1.7B")
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-ASR-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "JackyHoCL/qwen3-asr-yue-cantonese-lora")
model = model.to("cuda").eval()

# The outer wrapper class has no `forward`; route it to thinker.forward
# (same patch as the official Qwen3-ASR fine-tuning script):
def _patch(outer):
    outer_cls = type(outer)
    if getattr(outer_cls.__dict__.get("forward", None), "_patched", False):
        return
    def forward(self, *args, **kwargs):
        return self.thinker.forward(*args, **kwargs)
    forward._patched = True
    outer_cls.forward = forward
_patch(model)

audio = sf.read("clip.wav", dtype="float32")[0]
if audio.ndim > 1:
    audio = audio.mean(axis=1)

# prefix: empty system + audio user turn, with forced Cantonese language tag
prefix = processor.apply_chat_template(
    [{"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": "placeholder"}]}],
    add_generation_prompt=True, tokenize=False)[0] + "language Cantonese<asr_text>"
inp = processor(text=[prefix], audio=[audio], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=512)
new = out[0][inp["input_ids"].shape[1]:]
print(processor.batch_decode([new], skip_special_tokens=True)[0])

Usage β€” vLLM

Official stack: pip install "qwen-asr[vllm]" (extra pins a vLLM release; newer vLLM such as 0.30 also works with the bundled qwen_asr.core.vllm_backend model implementation).

Serving this LoRA adapter (recommended)

qwen-asr-serve is the official wrapper around vllm serve with the Qwen3-ASR model registry pre-loaded. Serve the base model with the adapter as a vLLM LoRA module (rank 16 β†’ --max-lora-rank 32):

vllm serve Qwen/Qwen3-ASR-1.7B \
    --enable-lora --max-lora-rank 32 \
    --lora-modules qwen3-asr-yue-cantonese=JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-lora \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.8 \
    --port 8000

Clients choose the adapter or the base by model name:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-asr-yue-cantonese",
    "messages": [
      {"role": "system", "content": ""},
      {"role": "user", "content": [
        {"type": "audio_url",
         "audio_url": {"url": "https://example.com/clip.wav"}}
      ]}
    ],
    "max_tokens": 512,
    "temperature": 0
  }'

The free-form response starts with the trained language Cantonese<asr_text> prefix β€” strip it to get the plain transcript (e.g. qwen_asr.parse_asr_output or a regex on the leading language …<asr_text> run).

Notes

  • Repo role: release builds. Dev / test / preview: JackyHoCL/qwen3-asr-yue-lora-preview.
  • The audio tower (perceiver + conv stem) is frozen; LoRA covers LLM self-attention + MLP projections only.
  • Eval WER/CER: Levenshtein over character tokens, normalized by reference length, punctuation stripped on both sides.
  • Lineage: v1 (8 h) β†’ v2 (110 h) β†’ v3 (103.7 h refreshed) β†’ v4 (227.9 h, 3 epochs, best-of-3 checkpoint).
Downloads last month
49
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-lora

Adapter
(28)
this model