Instructions to use scottgeng00/qwen3-0.6b-dclm-20b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use scottgeng00/qwen3-0.6b-dclm-20b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="scottgeng00/qwen3-0.6b-dclm-20b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b")
model = AutoModelForCausalLM.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use scottgeng00/qwen3-0.6b-dclm-20b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "scottgeng00/qwen3-0.6b-dclm-20b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgeng00/qwen3-0.6b-dclm-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/scottgeng00/qwen3-0.6b-dclm-20b

SGLang

How to use scottgeng00/qwen3-0.6b-dclm-20b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "scottgeng00/qwen3-0.6b-dclm-20b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgeng00/qwen3-0.6b-dclm-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "scottgeng00/qwen3-0.6b-dclm-20b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scottgeng00/qwen3-0.6b-dclm-20b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use scottgeng00/qwen3-0.6b-dclm-20b with Docker Model Runner:
```
docker model run hf.co/scottgeng00/qwen3-0.6b-dclm-20b
```

Qwen3-0.6B architecture, trained from scratch on 20B DCLM tokens

A 0.6B-parameter (595.8M, tied embeddings) model with the Qwen3-0.6B architecture and tokenizer, pretrained from random initialization on ~20.0B tokens of DCLM (the dclm source of the OLMo 2 Dolmino mix) with lingua. This is a research baseline, not an instruction-tuned model.

Training

Architecture: Qwen3-0.6B (28 layers, d=1024, 16 q / 8 kv heads, head_dim 128, SwiGLU 3072, QK-norm, RoPE theta 1e6, tied embeddings). Vocab = len(tokenizer) = 151669.
Data: DCLM only, documents separated by <|endoftext|>, packed to 4096 tokens with document masking (no cross-document attention).
19,074 steps x 256 sequences x 4096 tokens = 20.0B tokens; AdamW (lr 2.65e-3, betas 0.9/0.95, wd 0.05), 954 warmup steps, cosine decay to 1% of peak; bf16; 32 H200 GPUs.

Evaluation (final checkpoint)

OLMES (v0.4.13 port), base-model formats:

task	metric	score
SciQ (RC, 5-shot)	acc	0.798
ARC-Easy (RC, 5-shot)	acc	0.688
MMLU (RC, 5-shot, macro over 57 subjects)	acc	0.336
GSM8K (8-shot CoT)	exact match	0.020
TriviaQA (5-shot)	F1	0.254
GSM8K	gold-answer bits/byte	0.666
MBPP (3-shot)	gold-solution bits/byte	0.912

Held-out NLL per token on Dolmino sources (1000 docs each): dclm 2.610, math 1.632, flan 3.517, wiki 2.645, pes2o 2.763, stackexchange 2.773.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b")
model = AutoModelForCausalLM.from_pretrained("scottgeng00/qwen3-0.6b-dclm-20b", torch_dtype="bfloat16")

Downloads last month: 18

Safetensors

Model size

0.6B params

Tensor type

BF16