Instructions to use second-state/Octopus-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use second-state/Octopus-v2-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="second-state/Octopus-v2-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("second-state/Octopus-v2-GGUF")
model = AutoModelForCausalLM.from_pretrained("second-state/Octopus-v2-GGUF")

llama-cpp-python

How to use second-state/Octopus-v2-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="second-state/Octopus-v2-GGUF",
	filename="Octopus-v2-Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use second-state/Octopus-v2-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf second-state/Octopus-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf second-state/Octopus-v2-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf second-state/Octopus-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf second-state/Octopus-v2-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf second-state/Octopus-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf second-state/Octopus-v2-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf second-state/Octopus-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf second-state/Octopus-v2-GGUF:Q4_K_M

Use Docker

docker model run hf.co/second-state/Octopus-v2-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use second-state/Octopus-v2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "second-state/Octopus-v2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Octopus-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/second-state/Octopus-v2-GGUF:Q4_K_M

SGLang

How to use second-state/Octopus-v2-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "second-state/Octopus-v2-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Octopus-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "second-state/Octopus-v2-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Octopus-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use second-state/Octopus-v2-GGUF with Ollama:
```
ollama run hf.co/second-state/Octopus-v2-GGUF:Q4_K_M
```

Unsloth Studio

How to use second-state/Octopus-v2-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for second-state/Octopus-v2-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for second-state/Octopus-v2-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for second-state/Octopus-v2-GGUF to start chatting

Docker Model Runner
How to use second-state/Octopus-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/second-state/Octopus-v2-GGUF:Q4_K_M
```

Lemonade

How to use second-state/Octopus-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull second-state/Octopus-v2-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Octopus-v2-GGUF-Q4_K_M

List all available models

lemonade list

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:476b2976ac703ad133e32d6aa0093b0578ccdfe6142fcb4e2ae70805b7171132
-size 1157962112

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0bfe3fbf7db18148474b27229009cabc2c9c4b85f9e0d75835e572f1983e658c
-size 1465629056

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:87dd5255af4f2ddcb84aab91355bc3b3b6dc8ad29f36c49a803199941093b517
-size 1383840128

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8cbdecf73caddbe0d69d019081d55f2efb101cbe46f082577d245391566ed2b
-size 1288018304

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3647fa1db4fbe3feaf3a09c86e63519320aaf11a4b3f8c1fe0650cb758ea80fc
-size 1551227264

second-state
/

Octopus-v2-GGUF

Update models

 version https://git-lfs.github.com/spec/v1
+oid sha256:6c3e4fc3cb88c6a001a78b5663a857b511d1bf9f66fc61f057af3b19d2fb18ae
+size 1157962080

 version https://git-lfs.github.com/spec/v1
+oid sha256:b3d59aa88031546898953a4cebcd3f68248be6309a70ad0764f59fe86429167c
+size 1465629024

 version https://git-lfs.github.com/spec/v1
+oid sha256:cabf93a06e9fd4f3769fca00eff3be6a8953a079c585c6afb1193799088b475f
+size 1383840096

 version https://git-lfs.github.com/spec/v1
+oid sha256:bc401c0863fb2ab87fe558ffcd4cb3df00e0e990236273c844b9579c2fb5e548
+size 1288018272

 version https://git-lfs.github.com/spec/v1
+oid sha256:d24db1d1c2020d8e8dd828dfcc74505b635cfd129af6be7fa3b71cd8c249a225
+size 1551227232

 version https://git-lfs.github.com/spec/v1
+oid sha256:d8387c6fa6324016e30e6bfcf7840673b48bbd9ac592dc89d699c2b4ed89228c
+size 1630300512

 version https://git-lfs.github.com/spec/v1
+oid sha256:7713d7a5d926c04cf32fda0f2b7dcb997e08143637e422e04ef4c69a1f52910b
+size 1559877984

 version https://git-lfs.github.com/spec/v1
+oid sha256:e7d9ef87d0622095fa32975c8c537c00f8e4ca8754df5e1682bc52b61a58132b
+size 1798953312

 version https://git-lfs.github.com/spec/v1
+oid sha256:7b6ffe3be3f5bbae0cc79e8f83d0b9b3b1a41e83056c82b22f0e447f48ce23af
+size 1839688032

 version https://git-lfs.github.com/spec/v1
+oid sha256:b98a147902545ce3e870459874adec748a2c40df37a0013a320138a19381eb8a
+size 2062162272

 version https://git-lfs.github.com/spec/v1
+oid sha256:a85db45807a0d26b2c14753cea10f947a26196bde3770c95a2d0688b1bd6c127
+size 2669118464

 version https://git-lfs.github.com/spec/v1
+oid sha256:8e14ed8a6cb85d1774cc3402122e35ced6284ee01b9f3b25f7b08c22864e8401
+size 10030909664