Publish calibrated LoRA detector and model card

README.md

@@ -0,0 +1,153 @@
+---
+language:
+  - ko
+license: gemma
+library_name: peft
+pipeline_tag: text-classification
+base_model: google/embeddinggemma-300m
+base_model_relation: adapter
+tags:
+  - text-classification
+  - ai-text-detection
+  - authorship-analysis
+  - korean
+  - fiction
+  - stylometry
+---
+
+# ⚠️ Do not use this model as evidence of AI authorship.
+
+**Munche-768-AI-Detector was built for dataset triage and proof-of-concept research. It cannot establish plagiarism, copyright infringement, misconduct, or whether a person or an AI wrote a text. Its output can be wrong.**
+
+# Munche-768-AI-Detector
+
+<p align="center">
+  <img src="./assets/baragi-ai.png" width="128" alt="Baragi AI">
+</p>
+
+Munche-768-AI-Detector classifies Korean genre-fiction passages as `human`, `uncertain`, or `llm`. It starts from [Munche-768](https://huggingface.co/Baragi-AI/Munche-768), then jointly tunes LoRA weights in the top four Transformer layers and a 769-parameter linear classifier.
+
+## Overall test result
+
+The sealed test combines 277 human passages and 256 LLM passages from the independent-generation and content-preserving rewrite evaluations.
+
+| Metric | Result |
+|---|---:|
+| AUROC | **98.59%** |
+| Binary accuracy | **94.00%** |
+| Balanced accuracy | **93.91%** |
+| Human recall | **96.03%** |
+| LLM recall | **91.80%** |
+| Human false-positive rate | **3.97%** |
+
+<p align="center">
+  <img src="./assets/overall-confusion-matrix.svg" width="900" alt="Overall binary confusion matrix across independent generation and content-preserving rewrites">
+</p>
+
+## Independent-generation test
+
+The sealed test contains 87 passages from human-written novels and 66 passages written directly by 11 language-model families. Content-preserving rewrites are excluded from this evaluation.
+
+| Metric | Result |
+|---|---:|
+| Binary accuracy | **100.00%** |
+| Balanced accuracy | **100.00%** |
+| Human recall | **100.00%** |
+| LLM recall | **100.00%** |
+| Human false-positive rate | **0.00%** |
+
+<p align="center">
+  <img src="./assets/independent-confusion-matrix.svg" width="900" alt="Binary confusion matrix for independently written human and LLM fiction">
+</p>
+
+## Three-way decision
+
+The thresholds were selected on validation data only. The classification score is not a calibrated probability that a passage was written by AI. Results below cover the full sealed test.
+
+| Score | Output |
+|---:|---|
+| `≤ 0.5000` | Human |
+| `0.5000 < score < 0.8697` | Uncertain |
+| `≥ 0.8697` | LLM |
+
+| Metric | Result |
+|---|---:|
+| Coverage | **93.62%** |
+| Accuracy among classified passages | **95.19%** |
+| Human classified as LLM | **1.08%** |
+| Human classified as uncertain | **2.89%** |
+| LLM classified as human | **8.20%** |
+| LLM classified as uncertain | **10.16%** |
+
+Within the independent-generation subset, all 87 human passages received a human decision. Of the 66 LLM passages, 64 received an LLM decision and two were uncertain.
+
+<p align="center">
+  <img src="./assets/three-way-outcomes.svg" width="900" alt="Human, uncertain, and LLM outcomes for the independent-generation test">
+</p>
+
+## Content-preserving rewrite test
+
+This test contains 190 human passages and 190 LLM rewrites that preserve the source content. It is harder than distinguishing independently written human and LLM fiction.
+
+| Metric | Binary decision | Three-way decision |
+|---|---:|---:|
+| AUROC | **97.47%** | N/A |
+| Balanced accuracy | **91.58%** | N/A |
+| Human recall | **94.21%** | N/A |
+| LLM recall | **88.95%** | N/A |
+| Human classified as LLM | 5.79% | **1.58%** |
+| LLM classified as human | 11.05% | **11.05%** |
+| Coverage | N/A | **91.58%** |
+| Accuracy among classified passages | N/A | **93.10%** |
+
+<p align="center">
+  <img src="./assets/benchmark-comparison.svg" width="900" alt="Performance comparison between independent generation and content-preserving rewrites">
+</p>
+
+## Input length
+
+Use passages between **384 and 2,048 EmbeddingGemma tokens**. Inputs shorter than 384 tokens are not supported as stable operating inputs. Inputs longer than 2,048 tokens must be divided into separate windows before classification.
+
+The training and evaluation corpora covered short rewrite passages near 400 tokens and independent fiction passages near the 2,048-token model limit. Document-level aggregation across multiple windows has not been calibrated.
+
+## Usage
+
+Access to the gated EmbeddingGemma base model is required.
+
+```bash
+pip install torch numpy sentence-transformers peft safetensors
+```
+
+```python
+from inference import MuncheAIDetector
+
+detector = MuncheAIDetector(".")
+result = detector.predict(korean_fiction_passage)
+
+print(result)
+# {"label": "human" | "uncertain" | "llm", "score": float, "tokens": int}
+```
+
+## Training data
+
+| Split | Human | LLM |
+|---|---:|---:|
+| Train | 1,315 | 1,135 |
+| Validation | 290 | 275 |
+| Test | 277 | 256 |
+
+The training set combines human-written Korean genre fiction, independently generated LLM fiction, and content-preserving LLM rewrites. GPT-5.6 Sol Medium contributes 24 training passages and six validation passages. No Sol Medium passage was added to test.
+
+The detector was initialized from Munche-768. Only LoRA weights in Transformer layers 20-23 and the linear classifier were updated. The selected checkpoint is step 275. A preservation loss limited movement away from the original Munche-768 embedding during tuning.
+
+Raw human fiction is not distributed with this repository.
+
+## Limitations
+
+- The model was trained and evaluated on Korean genre fiction.
+- Generalization to language-model families absent from training remains unknown.
+- The model cannot identify text jointly written or substantially edited by humans and AI.
+
+## License
+
+Munche-768-AI-Detector is derived from `google/embeddinggemma-300m` and Munche-768. Use is subject to the Gemma license and the access terms of the gated base model.

adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/embeddinggemma-300m",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": false,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "down_proj",
+    "v_proj",
+    "up_proj",
+    "o_proj",
+    "k_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "FEATURE_EXTRACTION",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8802f80c3dc24b370dc6cb2937ac559475220ca4d32c7ce34a5664432cc4598f
+size 33465560

calibration.json

@@ -0,0 +1,78 @@
+{
+  "selection": {
+    "split": "validation",
+    "human_as_ai_max": 0.05,
+    "human_boundary": "fixed at the binary threshold"
+  },
+  "thresholds": {
+    "human_max": 0.5,
+    "ai_min": 0.8696600198745728
+  },
+  "validation": {
+    "combined": {
+      "samples": 565,
+      "coverage": 0.9079646017699115,
+      "covered_accuracy": 0.9551656920077972,
+      "human_as_ai": 0.04827586206896552,
+      "human_uncertain": 0.08275862068965517,
+      "ai_as_human": 0.03272727272727273,
+      "ai_uncertain": 0.10181818181818182
+    },
+    "rewrite": {
+      "samples": 406,
+      "coverage": 0.8768472906403941,
+      "covered_accuracy": 0.9353932584269663,
+      "human_as_ai": 0.06896551724137931,
+      "human_uncertain": 0.11330049261083744,
+      "ai_as_human": 0.04433497536945813,
+      "ai_uncertain": 0.1330049261083744
+    },
+    "independent": {
+      "samples": 153,
+      "coverage": 0.9934640522875817,
+      "covered_accuracy": 1.0,
+      "human_as_ai": 0.0,
+      "human_uncertain": 0.011494252873563218,
+      "ai_as_human": 0.0,
+      "ai_uncertain": 0.0
+    },
+    "sol_medium": {
+      "samples": 6,
+      "coverage": 0.8333333333333334,
+      "covered_accuracy": 1.0,
+      "human_as_ai": null,
+      "human_uncertain": null,
+      "ai_as_human": 0.0,
+      "ai_uncertain": 0.16666666666666666
+    }
+  },
+  "test": {
+    "combined": {
+      "samples": 533,
+      "coverage": 0.9362101313320825,
+      "covered_accuracy": 0.9519038076152304,
+      "human_as_ai": 0.010830324909747292,
+      "human_uncertain": 0.02888086642599278,
+      "ai_as_human": 0.08203125,
+      "ai_uncertain": 0.1015625
+    },
+    "rewrite": {
+      "samples": 380,
+      "coverage": 0.9157894736842105,
+      "covered_accuracy": 0.9310344827586207,
+      "human_as_ai": 0.015789473684210527,
+      "human_uncertain": 0.042105263157894736,
+      "ai_as_human": 0.11052631578947368,
+      "ai_uncertain": 0.12631578947368421
+    },
+    "independent": {
+      "samples": 153,
+      "coverage": 0.9869281045751634,
+      "covered_accuracy": 1.0,
+      "human_as_ai": 0.0,
+      "human_uncertain": 0.0,
+      "ai_as_human": 0.0,
+      "ai_uncertain": 0.030303030303030304
+    }
+  }
+}

config.json

@@ -0,0 +1,22 @@
+{
+  "architecture": "EmbeddingGemma with Munche-768 initialization, detector-tuned LoRA, and linear head",
+  "base_encoder": "google/embeddinggemma-300m",
+  "initial_adapter": "Baragi-AI/Munche-768",
+  "embedding_dim": 768,
+  "labels": {
+    "0": "human",
+    "1": "llm"
+  },
+  "max_tokens": 2048,
+  "supported_token_range": [384, 2048],
+  "selected_step": 275,
+  "updated_lora_layers": [20, 21, 22, 23],
+  "trainable_lora_parameters": 1392640,
+  "trainable_head_parameters": 769,
+  "seed": 20260727,
+  "three_way_thresholds": {
+    "human_max": 0.5,
+    "ai_min": 0.8696600198745728,
+    "human_as_ai_validation_target": 0.05
+  }
+}

inference.py

@@ -0,0 +1,50 @@
+import json
+from pathlib import Path
+
+import numpy as np
+from peft import PeftModel
+from safetensors.numpy import load_file
+from sentence_transformers import SentenceTransformer
+
+
+class MuncheAIDetector:
+    def __init__(self, model_dir: str | Path = "."):
+        model_dir = Path(model_dir)
+        self.encoder = SentenceTransformer("google/embeddinggemma-300m")
+        self.encoder[0].auto_model = PeftModel.from_pretrained(
+            self.encoder[0].auto_model,
+            model_dir,
+        )
+        self.encoder.max_seq_length = 2048
+        head = load_file(model_dir / "linear_head.safetensors")
+        self.weight = head["linear.weight"].reshape(-1)
+        self.bias = float(head["linear.bias"].item())
+        calibration = json.loads(
+            (model_dir / "calibration.json").read_text(encoding="utf-8")
+        )
+        self.human_max = calibration["thresholds"]["human_max"]
+        self.ai_min = calibration["thresholds"]["ai_min"]
+
+    def predict(self, text: str) -> dict:
+        tokens = self.encoder.tokenizer.encode(text, add_special_tokens=False)
+        if not 384 <= len(tokens) <= 2048:
+            raise ValueError("Input must contain 384 to 2,048 EmbeddingGemma tokens.")
+        embedding = self.encoder.encode(
+            [text],
+            normalize_embeddings=True,
+            convert_to_numpy=True,
+        )[0]
+        logit = float(embedding @ self.weight + self.bias)
+        score = (
+            float(1.0 / (1.0 + np.exp(-logit)))
+            if logit >= 0
+            else float(np.exp(logit) / (1.0 + np.exp(logit)))
+        )
+        label = (
+            "human"
+            if score <= self.human_max
+            else "llm"
+            if score >= self.ai_min
+            else "uncertain"
+        )
+        return {"label": label, "score": score, "tokens": len(tokens)}

length_stats.json

@@ -0,0 +1,96 @@
+{
+  "tokenizer": "google/embeddinggemma-300m",
+  "model_max_tokens": 2048,
+  "corpora": {
+    "independent_human": {
+      "samples": 783,
+      "raw_tokens": {
+        "minimum": 1708,
+        "p05": 1906,
+        "median": 2094,
+        "p95": 2258,
+        "maximum": 3920
+      },
+      "effective_tokens": {
+        "minimum": 1708,
+        "p05": 1906,
+        "median": 2048,
+        "p95": 2048,
+        "maximum": 2048
+      },
+      "truncated_fraction": 0.7075351213282248
+    },
+    "independent_ai": {
+      "samples": 330,
+      "raw_tokens": {
+        "minimum": 891,
+        "p05": 1271,
+        "median": 1999,
+        "p95": 2333,
+        "maximum": 2425
+      },
+      "effective_tokens": {
+        "minimum": 891,
+        "p05": 1271,
+        "median": 1999,
+        "p95": 2048,
+        "maximum": 2048
+      },
+      "truncated_fraction": 0.4636363636363636
+    },
+    "rewrite_human": {
+      "samples": 1306,
+      "raw_tokens": {
+        "minimum": 377,
+        "p05": 390,
+        "median": 406,
+        "p95": 430,
+        "maximum": 785
+      },
+      "effective_tokens": {
+        "minimum": 377,
+        "p05": 390,
+        "median": 406,
+        "p95": 430,
+        "maximum": 785
+      },
+      "truncated_fraction": 0.0
+    },
+    "rewrite_ai": {
+      "samples": 1306,
+      "raw_tokens": {
+        "minimum": 240,
+        "p05": 326,
+        "median": 420,
+        "p95": 514,
+        "maximum": 680
+      },
+      "effective_tokens": {
+        "minimum": 240,
+        "p05": 326,
+        "median": 420,
+        "p95": 514,
+        "maximum": 680
+      },
+      "truncated_fraction": 0.0
+    },
+    "sol_medium": {
+      "samples": 30,
+      "raw_tokens": {
+        "minimum": 1737,
+        "p05": 1974,
+        "median": 2173,
+        "p95": 2379,
+        "maximum": 2474
+      },
+      "effective_tokens": {
+        "minimum": 1737,
+        "p05": 1974,
+        "median": 2048,
+        "p95": 2048,
+        "maximum": 2048
+      },
+      "truncated_fraction": 0.8333333333333334
+    }
+  }
+}

linear_head.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5e32168ee1bd4fabf22fbc3914c78e580bb420fec630b900690ab9f582e42e
+size 3220

metrics.json

@@ -0,0 +1,146 @@
+{
+  "model": "Munche-768-AI-Detector",
+  "selected_step": 275,
+  "splits": {
+    "train": {
+      "human": 1315,
+      "llm": 1135
+    },
+    "validation": {
+      "human": 290,
+      "llm": 275
+    },
+    "test": {
+      "human": 277,
+      "llm": 256
+    }
+  },
+  "binary_test": {
+    "combined": {
+      "auroc": 0.9859403203971119,
+      "balanced_accuracy": 0.9391287793321299,
+      "human_recall": 0.9602888086642599,
+      "ai_recall": 0.91796875,
+      "confusion_matrix_human_ai": [
+        [
+          266,
+          11
+        ],
+        [
+          21,
+          235
+        ]
+      ]
+    },
+    "rewrite": {
+      "auroc": 0.9746814404432134,
+      "balanced_accuracy": 0.9157894736842105,
+      "human_recall": 0.9421052631578948,
+      "ai_recall": 0.8894736842105263,
+      "confusion_matrix_human_ai": [
+        [
+          179,
+          11
+        ],
+        [
+          21,
+          169
+        ]
+      ]
+    },
+    "independent": {
+      "auroc": 1.0,
+      "balanced_accuracy": 1.0,
+      "human_recall": 1.0,
+      "ai_recall": 1.0,
+      "confusion_matrix_human_ai": [
+        [
+          87,
+          0
+        ],
+        [
+          0,
+          66
+        ]
+      ]
+    }
+  },
+  "three_way": {
+    "selection": {
+      "split": "validation",
+      "human_as_ai_max": 0.05,
+      "human_boundary": "fixed at the binary threshold"
+    },
+    "thresholds": {
+      "human_max": 0.5,
+      "ai_min": 0.8696600198745728
+    },
+    "validation": {
+      "combined": {
+        "samples": 565,
+        "coverage": 0.9079646017699115,
+        "covered_accuracy": 0.9551656920077972,
+        "human_as_ai": 0.04827586206896552,
+        "human_uncertain": 0.08275862068965517,
+        "ai_as_human": 0.03272727272727273,
+        "ai_uncertain": 0.10181818181818182
+      },
+      "rewrite": {
+        "samples": 406,
+        "coverage": 0.8768472906403941,
+        "covered_accuracy": 0.9353932584269663,
+        "human_as_ai": 0.06896551724137931,
+        "human_uncertain": 0.11330049261083744,
+        "ai_as_human": 0.04433497536945813,
+        "ai_uncertain": 0.1330049261083744
+      },
+      "independent": {
+        "samples": 153,
+        "coverage": 0.9934640522875817,
+        "covered_accuracy": 1.0,
+        "human_as_ai": 0.0,
+        "human_uncertain": 0.011494252873563218,
+        "ai_as_human": 0.0,
+        "ai_uncertain": 0.0
+      },
+      "sol_medium": {
+        "samples": 6,
+        "coverage": 0.8333333333333334,
+        "covered_accuracy": 1.0,
+        "human_as_ai": null,
+        "human_uncertain": null,
+        "ai_as_human": 0.0,
+        "ai_uncertain": 0.16666666666666666
+      }
+    },
+    "test": {
+      "combined": {
+        "samples": 533,
+        "coverage": 0.9362101313320825,
+        "covered_accuracy": 0.9519038076152304,
+        "human_as_ai": 0.010830324909747292,
+        "human_uncertain": 0.02888086642599278,
+        "ai_as_human": 0.08203125,
+        "ai_uncertain": 0.1015625
+      },
+      "rewrite": {
+        "samples": 380,
+        "coverage": 0.9157894736842105,
+        "covered_accuracy": 0.9310344827586207,
+        "human_as_ai": 0.015789473684210527,
+        "human_uncertain": 0.042105263157894736,
+        "ai_as_human": 0.11052631578947368,
+        "ai_uncertain": 0.12631578947368421
+      },
+      "independent": {
+        "samples": 153,
+        "coverage": 0.9869281045751634,
+        "covered_accuracy": 1.0,
+        "human_as_ai": 0.0,
+        "human_uncertain": 0.0,
+        "ai_as_human": 0.0,
+        "ai_uncertain": 0.030303030303030304
+      }
+    }
+  }
+}