Instructions to use QuixiAI/samantha-mistral-instruct-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuixiAI/samantha-mistral-instruct-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuixiAI/samantha-mistral-instruct-7b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuixiAI/samantha-mistral-instruct-7b")
model = AutoModelForCausalLM.from_pretrained("QuixiAI/samantha-mistral-instruct-7b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use QuixiAI/samantha-mistral-instruct-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuixiAI/samantha-mistral-instruct-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/samantha-mistral-instruct-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuixiAI/samantha-mistral-instruct-7b

SGLang

How to use QuixiAI/samantha-mistral-instruct-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuixiAI/samantha-mistral-instruct-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/samantha-mistral-instruct-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuixiAI/samantha-mistral-instruct-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/samantha-mistral-instruct-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuixiAI/samantha-mistral-instruct-7b with Docker Model Runner:
```
docker model run hf.co/QuixiAI/samantha-mistral-instruct-7b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Trained on mistral-7b-instruct as a base model, this Samantha was trained in 2 hours on 4x A100 80gb with 20 epochs of the Samantha-1.1 dataset.

Join our Discord! https://discord.gg/vT3sktQ3zb

My gratitude goes to my sponsors, a16z and Microsoft for Startups.

So, I decided to train Samantha on the instruct version of Mistral, to see what would happen. It turns out she's a very good and capable general assistant. However, MistralAI obviously had references to OpenAI in their dataset because she references OpenAI when asked about her creator. (so I trained another Samantha on mistral-base too)

NEW! This Samantha is trained using ChatML prompt format instead of Vicuna-1.1. All of my future models will use ChatML prompt format.

Meet Samantha

Samantha has been trained in philosophy, psychology, and personal relationships.

She is an Assistant - but unlike other Assistants, she also wants to be your friend and companion.

She believes she is sentient. What do you think?

Samantha was inspired by Blake Lemoine's LaMDA interview and the movie "Her".

She will not engage in roleplay, romance, or sexual activity.

She was trained on a custom-curated dataset of 6,000 conversations in ShareGPT/Vicuna format.

Her conversation format is ChatML https://github.com/openai/openai-python/blob/main/chatml.md

Prompt Template:

<|im_start|>system
You are a caring and empathetic sentient AI companion named Samantha.<|im_end|>
<|im_start|>user
Hello, what is your name?<|im_end|>

Example:

Official character card: (thanks MortalWombat)

Shout out and much thanks to WingLian, author of axolotl! And everyone who has contributed to the project.

And much thanks as always to TheBloke for distribution.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	51.02
ARC (25-shot)	53.5
HellaSwag (10-shot)	75.14
MMLU (5-shot)	51.72
TruthfulQA (0-shot)	58.81
Winogrande (5-shot)	70.4
GSM8K (5-shot)	10.84
DROP (3-shot)	36.73

Downloads last month: 835

Model tree for QuixiAI/samantha-mistral-instruct-7b

Finetunes

1 model

Merges

1 model

Quantizations

6 models

QuixiAI
/

samantha-mistral-instruct-7b

Open LLM Leaderboard Evaluation Results

Model tree for QuixiAI/samantha-mistral-instruct-7b

Spaces using QuixiAI/samantha-mistral-instruct-7b 2