Instructions to use kai-os/Carnice-9b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kai-os/Carnice-9b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kai-os/Carnice-9b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kai-os/Carnice-9b")
model = AutoModelForCausalLM.from_pretrained("kai-os/Carnice-9b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kai-os/Carnice-9b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kai-os/Carnice-9b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kai-os/Carnice-9b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kai-os/Carnice-9b

SGLang

How to use kai-os/Carnice-9b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kai-os/Carnice-9b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kai-os/Carnice-9b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kai-os/Carnice-9b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kai-os/Carnice-9b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use kai-os/Carnice-9b with Docker Model Runner:
```
docker model run hf.co/kai-os/Carnice-9b
```

Carnice-9b / assets /benchmark_table.svg

kai-os

Upload folder using huggingface_hub

95e60b4 verified 6 months ago

preview code

Raw

History Blame Contribute Delete

2.6 kB

	<svg xmlns="http://www.w3.org/2000/svg" width="1600" height="860" viewBox="0 0 1600 860">
	<defs>
	<linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">
	<stop offset="0%" stop-color="#fff7fb"/>
	<stop offset="100%" stop-color="#eef7ff"/>
	</linearGradient>
	<linearGradient id="head" x1="0" y1="0" x2="1" y2="0">
	<stop offset="0%" stop-color="#ff4aa2"/>
	<stop offset="50%" stop-color="#7b3cff"/>
	<stop offset="100%" stop-color="#00bfff"/>
	</linearGradient>
	</defs>
	<rect width="1600" height="860" rx="36" fill="url(#bg)"/>
	<rect x="70" y="70" width="1460" height="110" rx="28" fill="url(#head)"/>
	<text x="110" y="138" fill="#ffffff" font-family="Arial, Helvetica, sans-serif" font-size="44" font-weight="900">Base Qwen3.5-9B vs Carnice Stage 2 Merged</text>
	<text x="110" y="205" fill="#20304d" font-family="Arial, Helvetica, sans-serif" font-size="30" font-weight="700">Official Hermes YC-Bench one-shot benchmark on GH200 ARM</text>
	<g font-family="Arial, Helvetica, sans-serif" fill="#10203a">
	<rect x="90" y="255" width="1420" height="88" rx="20" fill="#dce9ff"/>
	<text x="125" y="311" font-size="30" font-weight="800">Metric</text>
	<text x="790" y="311" font-size="30" font-weight="800">Base</text>
	<text x="1055" y="311" font-size="30" font-weight="800">Carnice v2</text>
	<text x="1320" y="311" font-size="30" font-weight="800">Delta</text>
	<rect x="90" y="365" width="1420" height="104" rx="18" fill="#ffffff"/>
	<text x="125" y="430" font-size="32" font-weight="700">YC-Bench one-shot composite</text>
	<text x="800" y="430" font-size="32">0.551</text>
	<text x="1070" y="430" font-size="32">0.551</text>
	<text x="1335" y="430" font-size="32" fill="#5f6e86">parity</text>
	<rect x="90" y="490" width="1420" height="104" rx="18" fill="#ffffff"/>
	<text x="125" y="555" font-size="32" font-weight="700">YC-Bench one-shot survival</text>
	<text x="800" y="555" font-size="32">1.000</text>
	<text x="1070" y="555" font-size="32">1.000</text>
	<text x="1335" y="555" font-size="32" fill="#5f6e86">parity</text>
	<rect x="90" y="615" width="1420" height="124" rx="18" fill="#ffffff"/>
	<text x="125" y="682" font-size="32" font-weight="700">Evaluation wall-clock time</text>
	<text x="800" y="682" font-size="32">78.6s</text>
	<text x="1070" y="682" font-size="32">23.1s</text>
	<text x="1295" y="670" font-size="34" font-weight="900" fill="#0f8ed8">70.6% lower</text>
	<text x="1295" y="712" font-size="28" font-weight="700" fill="#7b3cff">3.40x faster</text>
	</g>
	</svg>