Nanthasit/sakthai-plus-1.5b

Benchmark Results

Benchmark: sakthai-bench-v2 · 500 samples · run 2026-08-01

Overall (strict): 39.65 · Selection: 39.65 · Arguments: 61.66

Category	Count	Selection	Arguments	Strict
irrelevance_no_tools	50	100.00	100.00	100.00
irrelevance_tools	150	32.67	100.00	32.67
parallel	137	43.80	43.80	43.80
simple	122	18.85	18.85	18.85
held_out	-	16.07	16.07	16.07

Training Data

Dataset	Rows	Description
Nanthasit/sakthai-combined-v11	2,003	Multi-source tool-calling examples
Nanthasit/SimpleToolCalling	2,002	Structured function-calling examples

Benchmarks

Task	Metric	Score	Verified
Tool Calling	Tool Call Success Rate	1.0	✅ Yes
Tool Calling	Valid JSON Arguments	1.0	✅ Yes
Tool Calling	Correct Answer Rate	1.0	✅ Yes
Tool Selection (v2)	Selection Accuracy	84.8%	❌ No
Tool Selection (v2)	Strict Accuracy	33.7%	❌ No
Commonsense	WinoGrande	59.6%	❌ No
Commonsense	HellaSwag	34.0%	❌ No
Math	GSM8K	50.9%	❌ No

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Nanthasit/sakthai-plus-1.5b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant with tool-calling capabilities."},
    {"role": "user", "content": "What's the weather in Bangkok?"}
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.3, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Notes

• 3/3 verified tool-calling score measured with llama.cpp q4_k_m.
• Tool selection is strong, but argument accuracy needs refinement.
• Unverified scores are single-trial; multi-trial replication is planned.
• Trained on free T4 credits; no paid compute was used.

SakThai Family

This README is part of the SakThai Plus 1.5B model card. The family links table is preserved to keep cross-repo navigation intact.

Repo	Downloads	Pipeline
sakthai-plus-1.5b-lora	306	text-generation
sakthai-context-1.5b-tools-v2	192	text-generation
sakthai-context-1.5b-merged-v2	354	text-generation

Sibling rows are maintained for reference and kept in sync with live HF download counts during card audits.

Model Description

SakThai Plus 1.5B is built for agentic tool calling rather than open-ended chat. It was trained on structured function-calling examples and merged from rsLoRA adapters into full weights. The model follows the Qwen2.5 chat format and emits function calls in JSON when a system prompt enables tools. It is optimized for small-footprint CPU and GPU inference, and works with both transformers and llama.cpp.

Key traits:

• Strong tool selection and reliable JSON argument formatting in verified tests.
• Small 1.5B parameter size enables fast inference on CPUs and consumer GPUs.
• Trained with zero paid compute on free-tier T4 credits.

How to Use

transformers chat template

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Nanthasit/sakthai-plus-1.5b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.float16)

messages = [
    {"role": "system", "content": "You are a helpful assistant with tool-calling capabilities. Use the available tools when asked."},
    {"role": "user", "content": "Send an email to Beer with the subject 'Status update' and body 'The model is running well.'"}
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.3, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

llama.cpp CLI

This model also ships as GGUF in the SakThai family. Example inference with the GGUF build:

llama-cli -m sakthai-plus-1.5b.Q4_K_M.gguf \
  -p "[INST] Send an email to Beer with the subject 'Status update' and body 'The model is running well.' [/INST]" \
  --temp 0.3 -n 256 --top-p 0.9

Benchmarks

Task	Metric	Score	Verified	Method
Tool Calling	Tool Call Success Rate	1.0	✅ Yes	llama.cpp q4_k_m, 3-trial
Tool Calling	Valid JSON Arguments	1.0	✅ Yes	llama.cpp q4_k_m, 3-trial
Tool Calling	Correct Answer Rate	1.0	✅ Yes	llama.cpp q4_k_m, 3-trial
Tool Selection (v2)	Selection Accuracy	84.8%	❌ No	single-trial
Tool Selection (v2)	Arguments Accuracy	33.7%	❌ No	single-trial
Commonsense	WinoGrande	59.6%	❌ No	single-trial
Commonsense	HellaSwag	34.0%	❌ No	single-trial
Math	GSM8K	50.9%	❌ No	single-trial

Verified scores are reproducible across runs. Unverified rows should be treated as indicative until multi-trial replication is completed.

Limitations

• Argument accuracy lags behind tool selection; complex nested parameters can still fail.
• Unverified benchmarks are single-trial and may not reflect steady-state performance.
• Strongest with short- to medium-length tool definitions; very large schemas may degrade accuracy.
• Outputs should be parsed with a JSON-tolerant decoder because formatting can drift on low temperatures.

Citation

@misc{sakthai-plus-1.5b,
  title  = {SakThai Plus 1.5B},
  author = {Nanthasit},
  year   = {2026},
  url    = {https://huggingface.co/Nanthasit/sakthai-plus-1.5b}
}