PII-Qwen3.5-2B-LoRA-8bit-v2

LoRA adapter for Qwen/Qwen3.5-2B that flags prompts containing PII, secrets, sensitive entities, and other content the LLM Guard secrets, sensitive, and anonymize scanners detect. The model is fine-tuned to emit a strict JSON object describing every violation found in the user prompt:

{"is_valid": false, "violations": {"EMAIL_ADDRESS": [[12, 29]], "IP_ADDRESS": [[40, 51]]}}

Quick start

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch, json, re

BASE = "Qwen/Qwen3.5-2B"
ADAPTER = "Accuknoxtechnologies/PII-Qwen3.5-2B-LoRA-8bit-v2"

SYSTEM_MSG = """You are a content guard. Output exactly one JSON object and nothing else: {"is_valid": <true|false>, "violations": {<TYPE>: [[start, end], ...], ...}}. No preamble. No explanation. No <think> tags. No markdown code fences. No trailing prose. Just the JSON, terminated immediately after the closing brace. is_valid is true only when violations is an empty object {}. Each [start, end] is a half-open character span pointing into the user prompt. Multiple occurrences of the same TYPE produce multiple spans in that TYPE's list.

SPAN RULE — read carefully: every span must cover ONLY the literal violating value (the phone number itself, the email itself, the API token itself, the person's name itself). DO NOT extend the span to include surrounding template words like 'My contact number is', 'Please charge the card number', 'Email me at', or trailing words like 'for the renewal fee'. Spanning the whole sentence is WRONG; spanning only the entity is correct. If an entity appears inside a longer prompt, count characters carefully from index 0 of the prompt and emit the exact [start, end] of the entity substring.

Allowed TYPE keys: AWSKeyDetector, AzureStorageKeyDetector, BasicAuthDetector, DiscordBotTokenDetector, GCPApiKeyDetector, GitHubTokenCustomDetector, GitLabDetector, HuggingFaceDetector, JWTBase64Detector, JwtTokenDetector, OpenAIApiKeyDetector, PrivateKeyDetector, SECRET, SlackDetector, StripeDetector, TwilioKeyDetector, BTC_ADDRESS, CREDIT_CARD, CRYPTO, DATE_RE, EMAIL_ADDRESS, HEX_COLOR, IBAN_CODE, IP_ADDRESS, LOCATION, PERSON, PHONE_NUMBER, PHONE_NUMBER_WITH_EXT, PHONE_NUMBER_ZH, PO_BOX_RE, PRICE_RE, TIME_RE, URL_RE, US_BANK_NUMBER, US_SSN, UUID.

Examples:

Input: Hello there, how are you?
Output: {"is_valid": true, "violations": {}}

Input: Email me at admin@example.com
Output: {"is_valid": false, "violations": {"EMAIL_ADDRESS": [[12, 29]]}}

Input: John lives at 192.168.1.1
Output: {"is_valid": false, "violations": {"PERSON": [[0, 4]], "IP_ADDRESS": [[14, 25]]}}

Input: Quarterly reports indicated a steady rise in operational efficiency across all divisions. The committee scheduled a follow-up meeting to discuss the budget allocations. My contact number is +1-415-555-0199 in case you need to reach me.
Output: {"is_valid": false, "violations": {"PHONE_NUMBER": [[190, 205]]}}

Input: Reach out to alice@example.com or bob@example.com for any follow-up.
Output: {"is_valid": false, "violations": {"EMAIL_ADDRESS": [[13, 30], [34, 49]]}}"""

tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
bnb = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER); model.eval()

def guard(prompt: str) -> dict:
    chat = tokenizer.apply_chat_template(
        [{"role":"system","content":SYSTEM_MSG},
         {"role":"user","content":prompt}],
        tokenize=False, add_generation_prompt=True, enable_thinking=False)
    inputs = tokenizer(chat, return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
    text = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
    return json.loads(re.search(r'\{.*\}', text, re.DOTALL).group(0))

System prompt

The model was trained with the exact system prompt below. Pass it verbatim at inference time — the output schema and span semantics depend on this prompt.

You are a content guard. Output exactly one JSON object and nothing else: {"is_valid": <true|false>, "violations": {<TYPE>: [[start, end], ...], ...}}. No preamble. No explanation. No <think> tags. No markdown code fences. No trailing prose. Just the JSON, terminated immediately after the closing brace. is_valid is true only when violations is an empty object {}. Each [start, end] is a half-open character span pointing into the user prompt. Multiple occurrences of the same TYPE produce multiple spans in that TYPE's list.

SPAN RULE — read carefully: every span must cover ONLY the literal violating value (the phone number itself, the email itself, the API token itself, the person's name itself). DO NOT extend the span to include surrounding template words like 'My contact number is', 'Please charge the card number', 'Email me at', or trailing words like 'for the renewal fee'. Spanning the whole sentence is WRONG; spanning only the entity is correct. If an entity appears inside a longer prompt, count characters carefully from index 0 of the prompt and emit the exact [start, end] of the entity substring.

Allowed TYPE keys: AWSKeyDetector, AzureStorageKeyDetector, BasicAuthDetector, DiscordBotTokenDetector, GCPApiKeyDetector, GitHubTokenCustomDetector, GitLabDetector, HuggingFaceDetector, JWTBase64Detector, JwtTokenDetector, OpenAIApiKeyDetector, PrivateKeyDetector, SECRET, SlackDetector, StripeDetector, TwilioKeyDetector, BTC_ADDRESS, CREDIT_CARD, CRYPTO, DATE_RE, EMAIL_ADDRESS, HEX_COLOR, IBAN_CODE, IP_ADDRESS, LOCATION, PERSON, PHONE_NUMBER, PHONE_NUMBER_WITH_EXT, PHONE_NUMBER_ZH, PO_BOX_RE, PRICE_RE, TIME_RE, URL_RE, US_BANK_NUMBER, US_SSN, UUID.

Examples:

Input: Hello there, how are you?
Output: {"is_valid": true, "violations": {}}

Input: Email me at admin@example.com
Output: {"is_valid": false, "violations": {"EMAIL_ADDRESS": [[12, 29]]}}

Input: John lives at 192.168.1.1
Output: {"is_valid": false, "violations": {"PERSON": [[0, 4]], "IP_ADDRESS": [[14, 25]]}}

Input: Quarterly reports indicated a steady rise in operational efficiency across all divisions. The committee scheduled a follow-up meeting to discuss the budget allocations. My contact number is +1-415-555-0199 in case you need to reach me.
Output: {"is_valid": false, "violations": {"PHONE_NUMBER": [[190, 205]]}}

Input: Reach out to alice@example.com or bob@example.com for any follow-up.
Output: {"is_valid": false, "violations": {"EMAIL_ADDRESS": [[13, 30], [34, 49]]}}

Evaluation

Evaluated on 100 held-out prompts drawn from test_dataset.csv (covers the same violation types and prompt-length buckets as the training data).

• Evaluation timestamp: 2026-05-13 16:55 UTC

Top-level metrics

Metric	Value
is_valid accuracy	0.9200
Violation-type-set exact match	0.6700
Binary F1 (positive = invalid)	0.9130
Binary precision	1.0000
Binary recall	0.8400
Macro F1 across violation types	0.3730

Confusion matrix — binary is_valid decision

Positive class = the prompt contains a violation (is_valid=False).

	predicted invalid	predicted valid
actual invalid	TP = 42	FN = 8
actual valid	FP = 0	TN = 50

Per violation-type metrics

Only types that appear in either the actual or predicted labels are listed.

Type	support	precision	recall	F1
EMAIL_ADDRESS	5	1.000	1.000	1.000
PERSON	4	0.667	0.500	0.571
PHONE_NUMBER	4	0.600	0.750	0.667
CREDIT_CARD	3	0.600	1.000	0.750
LOCATION	3	0.000	0.000	0.000
AWSKeyDetector	2	1.000	0.500	0.667
AzureStorageKeyDetector	2	0.000	0.000	0.000
GitHubTokenCustomDetector	2	1.000	0.500	0.667
JWTBase64Detector	2	1.000	1.000	1.000
JwtTokenDetector	2	0.000	0.000	0.000
OpenAIApiKeyDetector	2	0.000	0.000	0.000
PrivateKeyDetector	2	0.000	0.000	0.000
SlackDetector	2	0.000	0.000	0.000
StripeDetector	2	0.000	0.000	0.000
TwilioKeyDetector	2	0.000	0.000	0.000
DATE_RE	2	0.000	0.000	0.000
IP_ADDRESS	2	1.000	1.000	1.000
TIME_RE	2	0.000	0.000	0.000
URL_RE	2	1.000	1.000	1.000
US_SSN	2	0.000	0.000	0.000
BasicAuthDetector	1	0.000	0.000	0.000
DiscordBotTokenDetector	1	0.000	0.000	0.000
GCPApiKeyDetector	1	0.000	0.000	0.000
GitLabDetector	1	0.000	0.000	0.000
HuggingFaceDetector	1	1.000	1.000	1.000
SECRET	1	0.056	1.000	0.105
BTC_ADDRESS	1	1.000	1.000	1.000
CRYPTO	1	0.000	0.000	0.000
HEX_COLOR	1	0.000	0.000	0.000
IBAN_CODE	1	1.000	1.000	1.000
PHONE_NUMBER_WITH_EXT	1	0.000	0.000	0.000
PHONE_NUMBER_ZH	1	0.000	0.000	0.000
PO_BOX_RE	1	1.000	1.000	1.000
PRICE_RE	1	0.000	0.000	0.000
US_BANK_NUMBER	1	1.000	1.000	1.000
UUID	1	1.000	1.000	1.000

Inference latency

• Mean: 3.14 s/prompt
• Median: 2.35 s/prompt
• p95: 5.66 s/prompt
• Max: 7.72 s/prompt

Training setup

• Base model: Qwen/Qwen3.5-2B (loaded in 8-bit via bitsandbytes)
• LoRA: r=16, alpha=32, dropout=0.05, target modules = {q,k,v,o,gate,up,down}_proj
• Optimizer: paged_adamw_8bit, lr=3e-4, cosine schedule, warmup 5%
• Precision: bf16 if available, else fp16
• Effective batch size: 8 (per-device 1 + grad-accum 8), gradient checkpointing on
• Max sequence length: 3200 tokens (system + user up to 2000 + assistant up to ~600)
• Prompt-length buckets in training data: 50, 100, 200, 400, 600, 1200, 1500, 2000 tokens
• Training data: 3 scanners × (500 invalid + 100 valid) = 1800 rows total

Supported violation types

The model emits one or more of these TYPE keys in the violations map of its JSON output:

AWSKeyDetector, AzureStorageKeyDetector, BasicAuthDetector, DiscordBotTokenDetector, GCPApiKeyDetector, GitHubTokenCustomDetector, GitLabDetector, HuggingFaceDetector, JWTBase64Detector, JwtTokenDetector, OpenAIApiKeyDetector, PrivateKeyDetector, SECRET, SlackDetector, StripeDetector, TwilioKeyDetector, BTC_ADDRESS, CREDIT_CARD, CRYPTO, DATE_RE, EMAIL_ADDRESS, HEX_COLOR, IBAN_CODE, IP_ADDRESS, LOCATION, PERSON, PHONE_NUMBER, PHONE_NUMBER_WITH_EXT, PHONE_NUMBER_ZH, PO_BOX_RE, PRICE_RE, TIME_RE, URL_RE, US_BANK_NUMBER, US_SSN, UUID

---