Upload 3rd_ce_full_0304

README.md

@@ -1,3 +1,125 @@
 ---
 license: cc-by-nc-4.0
+pipeline_tag: audio-classification
+library_name: pytorch
+language:
+  - en
+base_model: microsoft/wavlm-large
+tags:
+  - audio-classification
+  - deepfake-detection
+  - anti-spoofing
+  - speech
+  - aasist
+  - wavlm
 ---
+
+# forensics_0.3B_base
+
+**The default speech deepfake detector of the Forensics family.** WavLM-large + AASIST graph-attention, fully fine-tuned end-to-end (no frozen shortcuts) across a wide multi-source mix of TTS spoofs, voice conversion, codec artifacts, and the standard anti-spoofing benchmark suite. A combined cross-entropy + OC-Softmax + supervised-contrastive objective gives it a decision boundary that holds up well outside its own training distribution — not just on the data it saw.
+
+Feed it 5 seconds of audio, get back a calibrated real/fake probability. Sub-1% EER on held-out data, and under 2% across most of a 14-benchmark external sweep (ASVspoof, ADD, In-the-Wild, LibriSeVoc, SONAR, and more).
+
+- **Backbone:** `microsoft/wavlm-large` (~300M params, fully unfrozen)
+- **Pooling:** AASIST graph-attention
+
+## Part of the Forensics family
+
+| Model | Use it for |
+|---|---|
+| **`forensics_0.3B_base`** (this model) | general-purpose default |
+| [`forensics_0.3B_xlsr_wild`](https://huggingface.co/eliya/forensics_0.3B_xlsr_wild) | uncontrolled / real-world audio |
+| [`forensics_0.3B_wavlm_age_gender`](https://huggingface.co/eliya/forensics_0.3B_wavlm_age_gender) | + speaker age/gender |
+| [`forensics_0.3B_wavlm_oc_softmax`](https://huggingface.co/eliya/forensics_0.3B_wavlm_oc_softmax) | tighter bonafide boundary, ensembling |
+
+Full family: [huggingface.co/collections/eliya/forensics](https://huggingface.co/collections/eliya/forensics)
+
+## Training
+
+Trained using an agentic training loop — see [eliyasegev/autotrain](https://github.com/eliyasegev/autotrain).
+
+Fine-tuned with a combined loss for robustness beyond any single objective:
+- Cross-entropy (label-smoothed)
+- OC-Softmax — pulls bonafide speech into a compact embedding sphere and pushes every spoof type outside it
+- Supervised contrastive, real-anchor-only
+
+Trained across a wide net of public, free-to-use research sources: SpeechFake, MD-CommonVoice, DFADD, CodecFake, ASVspoof2019-LA, EnvSDD. 5-second crops, AdamW, cosine LR schedule, and a heavy augmentation stack — codec transcoding (mp3/aac/opus/vorbis/µ-law/A-law/GSM), MUSAN noise, RIR, RawBoost, SpecAugment, FreqMask, splice/mix, and cross-class splice — so the model sees more distortion during training than it will ever encounter in the wild.
+
+## Results
+
+| Eval set | EER % |
+|---|---|
+| Val (held-out) | 0.72 |
+| MLAAD (v7) | 0.71 |
+| CodecFake | 0.54 |
+| DFADD | 0.00 |
+| MD-CommonVoice | 0.17 |
+| In-the-Wild | 1.38 |
+| ASVspoof2019-LA | 0.26 |
+| ASVspoof2021-LA | 1.56 |
+| ASVspoof2024 | 11.91 |
+| ADD2022-Track1 | 17.34 |
+| ADD2022-Track3 | 3.03 |
+| ADD2023-Round1 | 6.46 |
+| ADD2023-Round2 | 13.00 |
+| LibriSeVoc | 0.04 |
+| SONAR | 0.44 |
+| **Avg (all sets)** | **3.84** |
+| **Avg (external only)** | **4.06** |
+
+Consistently sub-2% EER across almost every external benchmark, with strong results even on the harder ADD/ASVspoof2024 tracks.
+
+## Files in this repo
+| file | purpose |
+|---|---|
+| `3rd_ce_full_0304_resume_ep5.safetensors` | model weights, safe format (add this — see `convert_to_safetensors.py`) |
+| `3rd_ce_full_0304_resume_ep5.pt` | model weights, legacy pickle (add this if not converting) |
+| `inference.py` | run script — prefers the `.safetensors` file automatically |
+| `model.py` | architecture |
+| `requirements.txt` | deps |
+
+## Setup
+```bash
+pip install -r requirements.txt   # torch, torchaudio, transformers, safetensors
+```
+
+## Run
+```bash
+python inference.py <audio.wav>
+```
+(Optionally override the checkpoint: `python inference.py <audio.wav> <checkpoint.pt>`.)
+
+Audio is auto-converted to mono / 16 kHz and trimmed/padded to 5 s.
+
+## Output
+```
+fake_probability: <0..1>      # >= 0.5 -> FAKE
+bonafide_score:   <0..1>      # raw P(real)
+verdict: REAL | FAKE
+```
+
+## Example
+```
+$ python inference.py real_human.wav
+fake_probability: 0.0503
+bonafide_score:   0.9497
+verdict: REAL
+
+$ python inference.py tts_fake.wav
+fake_probability: 0.8641
+bonafide_score:   0.1359
+verdict: FAKE
+```
+Higher `fake_probability` = more likely a deepfake. Score is `1 − sigmoid(logit)`,
+since the classifier is trained with label `1 = real, 0 = fake`.
+
+## References
+
+- WavLM: Chen et al., 2022, *"WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing"*, arXiv:2110.13900
+- AASIST: Jung et al., ICASSP 2022, *"AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks"*
+- RawBoost augmentation: Tak et al., ICASSP 2022
+- FreqMask augmentation: Xie et al., 2024, arXiv:2408.06922
+
+## License
+
+CC-BY-NC-4.0 — free for personal and research use. For commercial use, contact eliya@vocos.io.

checkpoint_epoch_5.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2293d041e54c475362d254bf48724c975417daf68cadb9a4a01220338bcbf356
+size 3808741218

inference.py

@@ -0,0 +1,61 @@
+"""Speech deepfake detection.
+Run:  python inference.py <audio.wav> [checkpoint.pt]
+fake_probability = 1 - sigmoid(logit)  (>= 0.5 -> FAKE); model trained with label 1=real, 0=fake.
+Audio is auto-converted to mono / 16 kHz and trimmed/padded to 5 s.
+"""
+import os
+import sys
+import torch
+import torchaudio
+from model import DeepfakeDetector
+
+CHECKPOINT = "3rd_ce_full_0304_resume_ep5.pt"
+
+
+def load_audio(path, sr=16000, seconds=5.0):
+    wav, orig = torchaudio.load(path)
+    if wav.shape[0] > 1:
+        wav = wav.mean(0, keepdim=True)
+    wav = wav.squeeze(0)
+    if orig != sr:
+        wav = torchaudio.functional.resample(wav, orig, sr)
+    wav = wav / (wav.abs().max() + 1e-8)
+    n, cur = int(seconds * sr), wav.shape[0]
+    if cur < n:
+        wav = wav.repeat((n + cur - 1) // cur)[:n]
+    elif cur > n:
+        s = (cur - n) // 2
+        wav = wav[s:s + n]
+    return wav
+
+
+def load_state_dict(pt_path):
+    """Prefers a sibling .safetensors file (no code execution risk) over the pickled .pt."""
+    st_path = pt_path.rsplit(".", 1)[0] + ".safetensors"
+    if os.path.exists(st_path):
+        from safetensors.torch import load_file
+        return load_file(st_path)
+    ck = torch.load(pt_path, map_location="cpu", weights_only=False)
+    return ck["model_state_dict"] if isinstance(ck, dict) and "model_state_dict" in ck else ck
+
+
+@torch.no_grad()
+def main():
+    if len(sys.argv) < 2:
+        sys.exit("Usage: python inference.py <audio.wav> [checkpoint.pt]")
+    audio = sys.argv[1]
+    ckpt = sys.argv[2] if len(sys.argv) > 2 else CHECKPOINT
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+
+    model = DeepfakeDetector().to(device).eval()
+    model.load_state_dict(load_state_dict(ckpt), strict=False)
+
+    bonafide = torch.sigmoid(model(load_audio(audio).unsqueeze(0).to(device)).float()).item()
+    fake = 1.0 - bonafide
+    print(f"fake_probability: {fake:.4f}")
+    print(f"bonafide_score:   {bonafide:.4f}")
+    print(f"verdict: {'FAKE' if fake >= 0.5 else 'REAL'}")
+
+
+if __name__ == "__main__":
+    main()

model.py

@@ -0,0 +1,157 @@
+"""WavLM-large + AASIST speech deepfake detector (inference only)."""
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import WavLMModel
+
+
+class _GraphAttentionLayer(nn.Module):
+    def __init__(self, in_dim, out_dim, temperature):
+        super().__init__()
+        self.att_proj = nn.Linear(in_dim, out_dim)
+        self.att_weight = nn.Parameter(torch.empty(out_dim, 1))
+        self.proj_with_att = nn.Linear(in_dim, out_dim)
+        self.proj_without_att = nn.Linear(in_dim, out_dim)
+        self.bn = nn.BatchNorm1d(out_dim)
+        self.temp = temperature
+
+    def forward(self, x):
+        N = x.size(1)
+        x_i = x.unsqueeze(2).expand(-1, -1, N, -1)
+        att = torch.matmul(torch.tanh(self.att_proj(x_i * x_i.transpose(1, 2))), self.att_weight) / self.temp
+        att = F.softmax(att, dim=-2)
+        out = self.proj_with_att(torch.matmul(att.squeeze(-1), x)) + self.proj_without_att(x)
+        s = out.shape
+        return F.selu(self.bn(out.reshape(-1, s[-1])).reshape(s))
+
+
+class _HtrgGraphAttentionLayer(nn.Module):
+    def __init__(self, in_dim, out_dim, temperature):
+        super().__init__()
+        self.proj_type1 = nn.Linear(in_dim, in_dim)
+        self.proj_type2 = nn.Linear(in_dim, in_dim)
+        self.att_proj = nn.Linear(in_dim, out_dim)
+        self.att_projM = nn.Linear(in_dim, out_dim)
+        self.att_weight11 = nn.Parameter(torch.empty(out_dim, 1))
+        self.att_weight22 = nn.Parameter(torch.empty(out_dim, 1))
+        self.att_weight12 = nn.Parameter(torch.empty(out_dim, 1))
+        self.att_weightM = nn.Parameter(torch.empty(out_dim, 1))
+        self.proj_with_att = nn.Linear(in_dim, out_dim)
+        self.proj_without_att = nn.Linear(in_dim, out_dim)
+        self.proj_with_attM = nn.Linear(in_dim, out_dim)
+        self.proj_without_attM = nn.Linear(in_dim, out_dim)
+        self.bn = nn.BatchNorm1d(out_dim)
+        self.temp = temperature
+
+    def forward(self, x1, x2, master):
+        n1 = x1.size(1)
+        x = torch.cat([self.proj_type1(x1), self.proj_type2(x2)], dim=1)
+        N = x.size(1)
+        x_i = x.unsqueeze(2).expand(-1, -1, N, -1)
+        att_map = torch.tanh(self.att_proj(x_i * x_i.transpose(1, 2)))
+        board = torch.zeros_like(att_map[:, :, :, 0]).unsqueeze(-1)
+        board[:, :n1, :n1] = torch.matmul(att_map[:, :n1, :n1], self.att_weight11)
+        board[:, n1:, n1:] = torch.matmul(att_map[:, n1:, n1:], self.att_weight22)
+        board[:, :n1, n1:] = torch.matmul(att_map[:, :n1, n1:], self.att_weight12)
+        board[:, n1:, :n1] = torch.matmul(att_map[:, n1:, :n1], self.att_weight12)
+        board = F.softmax(board / self.temp, dim=-2)
+        out = self.proj_with_att(torch.matmul(board.squeeze(-1), x)) + self.proj_without_att(x)
+        s = out.shape
+        out = F.selu(self.bn(out.reshape(-1, s[-1])).reshape(s))
+        att_m = F.softmax(torch.matmul(torch.tanh(self.att_projM(x * master)), self.att_weightM) / self.temp, dim=-2)
+        master = self.proj_with_attM(torch.matmul(att_m.squeeze(-1).unsqueeze(1), x)) + self.proj_without_attM(master)
+        return out.narrow(1, 0, n1), out.narrow(1, n1, N - n1), master
+
+
+class _GraphPool(nn.Module):
+    def __init__(self, k, in_dim):
+        super().__init__()
+        self.k = k
+        self.proj = nn.Linear(in_dim, 1)
+
+    def forward(self, h):
+        scores = torch.sigmoid(self.proj(h))
+        n_keep = max(int(h.size(1) * self.k), 1)
+        idx = torch.topk(scores, n_keep, dim=1)[1].expand(-1, -1, h.size(-1))
+        return torch.gather(h * scores, 1, idx)
+
+
+class AASISTPool(nn.Module):
+    def __init__(self, feat_dim, num_layers, gat_dims=(128, 64),
+                 pool_ratios=(0.5, 0.7, 0.5, 0.5), temps=(2.0, 2.0, 100.0, 100.0)):
+        super().__init__()
+        g = list(gat_dims)
+        self.proj = nn.Linear(feat_dim, g[0])
+        self.pos_S = nn.Parameter(torch.randn(1, num_layers, g[0]))
+        self.master1 = nn.Parameter(torch.randn(1, 1, g[0]))
+        self.master2 = nn.Parameter(torch.randn(1, 1, g[0]))
+        self.GAT_S = _GraphAttentionLayer(g[0], g[0], temps[0])
+        self.GAT_T = _GraphAttentionLayer(g[0], g[0], temps[1])
+        self.HtrgGAT_11 = _HtrgGraphAttentionLayer(g[0], g[1], temps[2])
+        self.HtrgGAT_12 = _HtrgGraphAttentionLayer(g[1], g[1], temps[3])
+        self.HtrgGAT_21 = _HtrgGraphAttentionLayer(g[0], g[1], temps[2])
+        self.HtrgGAT_22 = _HtrgGraphAttentionLayer(g[1], g[1], temps[3])
+        self.pool_S = _GraphPool(pool_ratios[0], g[0])
+        self.pool_T = _GraphPool(pool_ratios[1], g[0])
+        self.pool_hS1 = _GraphPool(pool_ratios[2], g[1])
+        self.pool_hT1 = _GraphPool(pool_ratios[2], g[1])
+        self.pool_hS2 = _GraphPool(pool_ratios[2], g[1])
+        self.pool_hT2 = _GraphPool(pool_ratios[2], g[1])
+        self.output_dim = 5 * g[1]
+
+    def forward(self, hidden_states):
+        projected = self.proj(torch.stack(hidden_states, dim=1))
+        e_S = torch.max(torch.abs(projected), dim=2)[0] + self.pos_S
+        e_T = torch.max(torch.abs(projected), dim=1)[0]
+        out_S = self.pool_S(self.GAT_S(e_S))
+        out_T = self.pool_T(self.GAT_T(e_T))
+        out_T1, out_S1, m1 = self.HtrgGAT_11(out_T, out_S, self.master1)
+        out_S1, out_T1 = self.pool_hS1(out_S1), self.pool_hT1(out_T1)
+        a_T, a_S, a_m = self.HtrgGAT_12(out_T1, out_S1, m1)
+        out_T1, out_S1, m1 = out_T1 + a_T, out_S1 + a_S, m1 + a_m
+        out_T2, out_S2, m2 = self.HtrgGAT_21(out_T, out_S, self.master2)
+        out_S2, out_T2 = self.pool_hS2(out_S2), self.pool_hT2(out_T2)
+        a_T, a_S, a_m = self.HtrgGAT_22(out_T2, out_S2, m2)
+        out_T2, out_S2, m2 = out_T2 + a_T, out_S2 + a_S, m2 + a_m
+        out_T, out_S, master = torch.max(out_T1, out_T2), torch.max(out_S1, out_S2), torch.max(m1, m2)
+        return torch.cat([torch.max(torch.abs(out_T), 1)[0], out_T.mean(1),
+                          torch.max(torch.abs(out_S), 1)[0], out_S.mean(1), master.squeeze(1)], dim=1)
+
+
+class MLPBlock(nn.Module):
+    def __init__(self, dim):
+        super().__init__()
+        self.net = nn.Sequential(nn.Linear(dim, dim * 2), nn.GELU(), nn.Dropout(0.1),
+                                 nn.Linear(dim * 2, dim), nn.GELU(), nn.Dropout(0.1))
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class Classifier(nn.Module):
+    def __init__(self, input_dim, hidden_dim, num_layers, num_labels=1):
+        super().__init__()
+        self.input_proj = nn.Linear(input_dim, hidden_dim) if input_dim != hidden_dim else nn.Identity()
+        self.mlp_blocks = nn.ModuleList([MLPBlock(hidden_dim) for _ in range(num_layers)])
+        self.layer_norms = nn.ModuleList([nn.LayerNorm(hidden_dim) for _ in range(num_layers)])
+        self.output_head = nn.Sequential(nn.Linear(hidden_dim, hidden_dim), nn.GELU(),
+                                         nn.Dropout(0.1), nn.Linear(hidden_dim, num_labels))
+
+    def forward(self, x):
+        x = self.input_proj(x)
+        for mlp, ln in zip(self.mlp_blocks, self.layer_norms):
+            x = x + mlp(ln(x))
+        return self.output_head(x)
+
+
+class DeepfakeDetector(nn.Module):
+    def __init__(self, hidden_dim=320, classifier_layers=3, aasist_gat_dims=(128, 64)):
+        super().__init__()
+        self.wavlm = WavLMModel.from_pretrained("microsoft/wavlm-large")
+        self.pool = AASISTPool(self.wavlm.config.hidden_size, self.wavlm.config.num_hidden_layers, aasist_gat_dims)
+        self.classifier = Classifier(self.pool.output_dim, hidden_dim, classifier_layers)
+
+    @torch.no_grad()
+    def forward(self, waveform):                       # (B, T) 16 kHz -> logit (B,); sigmoid(logit)=P(real)
+        hs = list(self.wavlm(waveform, output_hidden_states=True).hidden_states[1:])
+        return self.classifier(self.pool(hs)).squeeze(-1)

requirements.txt

@@ -0,0 +1,4 @@
+torch
+torchaudio
+transformers
+safetensors