gipformer-callbot-vi-denoiser

Per-segment noise / speech classifier for Vietnamese callbot audio. Used to filter contaminated segments out of the unreviewed callbot pool before semi-supervised ASR fine-tuning (sister model: gipformer-callbot-vi-v4).

Operates on top of a 65M-param Vietnamese Zipformer encoder (icefall recipe). Input: K consecutive segment encodings (sliding window over a call). Output: a sigmoid probability that the LAST segment in the window is noise (vs speech).

Two variants

File	Encoder	Window K	Adapter params	Use when
classifier_lora.pt	Zipformer + LoRA adapters (rank 8, α 16) on self-attention in/out projs	8	304K	Best detection quality. Worth the extra adapter forward at inference.
classifier_frozen_k16.pt	Zipformer frozen	16	0	Simpler integration (no LoRA layer wrapping). Wider context window.

Both ship a ConvTransformerClassifier head (~1.6M params). Both were trained on the same mix: reviewed callbot (noise/speech tags) + an existing English UAT noise classification dataset.

Checkpoint contents

classifier_lora.pt

{
  "classifier_state": dict,   # 52 entries — classifier head weights
  "lora_state":       dict,   # 160 entries — LoRA A/B matrices for encoder
  "classifier_cfg":   dict,   # ConvTransformerClassifier init args
  "lora_rank":        8,
  "lora_alpha":       16,
  "window":           8,      # K
  "threshold":        0.5,    # default decision threshold on noise_prob
  "embedder":         "gipformer_rnnt",
  "pos_weight":       0.222,  # class imbalance weight used during training
}

classifier_frozen_k16.pt

{
  "model_state_dict": dict,   # 52 entries — classifier head weights
  "config":           dict,   # ConvTransformerClassifier init args
  "embedder":         "gipformer_rnnt",
  "window":           16,     # K
  "threshold":        0.5,
  "pos_weight":       0.222,
}

Usage (LoRA variant)

import torch
from denoiser.lora    import freeze_base, wrap_with_lora
from denoiser.model   import ConvTransformerClassifier
from gipformer.model.build import build_model

GIPFORMER_DIR = "models/gipformer-65M-rnnt"   # the v4 base; provides arch + epoch-35-avg-6.pt
device = torch.device("cuda")

ck = torch.load("classifier_lora.pt", map_location=device, weights_only=False)

# Encoder + LoRA wrap
enc, _ = build_model(GIPFORMER_DIR, device=device, use_ctc=False)
freeze_base(enc)
def _match(name, mod):
    if "self_attn" not in name: return False
    return name.endswith((".in_proj", ".out_proj"))
wrap_with_lora(enc, _match, rank=ck["lora_rank"], alpha=ck["lora_alpha"])
enc.to(device)

# Inject LoRA adapter weights
enc_sd = enc.state_dict()
for n, v in ck["lora_state"].items():
    if n in enc_sd:
        enc_sd[n].copy_(v.to(device))
    else:
        # tolerate prefix variants
        for sd_name in enc_sd:
            if sd_name.endswith(n):
                enc_sd[sd_name].copy_(v.to(device)); break
enc.eval()

# Classifier
clf = ConvTransformerClassifier(**ck["classifier_cfg"]).to(device)
clf.load_state_dict(ck["classifier_state"]); clf.eval()

K, thr = ck["window"], ck["threshold"]

Usage (frozen K=16 variant)

import torch
from denoiser.model   import ConvTransformerClassifier
from gipformer.model.build import build_model

device = torch.device("cuda")
enc, _ = build_model("models/gipformer-65M-rnnt", device=device, use_ctc=False)
enc.eval()

ck = torch.load("classifier_frozen_k16.pt", map_location=device, weights_only=False)
clf = ConvTransformerClassifier(**ck["config"]).to(device)
clf.load_state_dict(ck["model_state_dict"]); clf.eval()
K, thr = ck["window"], ck["threshold"]

Scoring pipeline (per call)

For each call: extract per-segment encoder embeddings ONCE (segments share encoder), then for each segment build a K-window of [max(0, i-K+1) ... i] encodings and classify. See denoiser/scripts/score_unreviewed.py in the source repo for the batched implementation (one encoder forward per call, GPU-batched window slicing).

noise_prob = 1 - sigmoid(classifier_logit[..., -1]). Default threshold 0.5; the v4 semi-sup pipeline used 0.3 (more permissive — favors keeping speech).

Training data

• Reviewed callbot segments tagged speech / noise (~3.5k speech + few hundred noise)
• An existing English UAT noise/speech classification dataset (~85% of training pool)

Class imbalance is ~78% speech / 22% noise — hence pos_weight=0.222 in BCEWithLogitsLoss (equivalent to weighting the speech class).

Limitations

• Trained primarily on callbot agent-side audio. Customer-side and out-of-domain audio may shift the noise_prob distribution; revalidate threshold per domain.
• Single-segment classification only; not a sequence labeler.
• Requires the Vietnamese gipformer-65M encoder (base) — not bundled here. Get it from the icefall recipe or the v4 model repo's loader path.