Instructions to use NoobieToPro/NYXIS-30B-A3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("NoobieToPro/NYXIS-30B-A3B-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Use Docker
docker model run hf.co/NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
- Llama new
- LM Studio
- Jan
- Ollama
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Ollama:
ollama run hf.co/NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
- Unsloth Desktop
- Pi
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Docker Model Runner:
docker model run hf.co/NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
- Lemonade
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Run and chat with the model
lemonade run user.NYXIS-30B-A3B-GGUF-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use NoobieToPro/NYXIS-30B-A3B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "NoobieToPro/NYXIS-30B-A3B-GGUF:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- ๐ NYXIS-30B-A3B โ Qwen3-VL MoE + Native Vision for the NYXIS Local Harness
- ๐๏ธ Custom Architecture & Multimodal Upgrade
- โก Why It Runs on a 6 GB RTX 4050 Laptop (2.73 GB VRAM)
- ๐ Real-World Hardware Benchmarks (RTX 4050 6 GB + 16 GB RAM, i5-13420H)
- โ๏ธ Empirical Head-to-Head Benchmarks (Measured on RTX 4050 6 GB + i5-13420H, 16 GB RAM)
- ๐งช Test 1 โ VRAM (per-process dedicated) & CPU working set
- ๐ฅ Test 2 โ Cold & warm code generation (identical 400-token HTML/JS todo-app prompt)
- ๐ง Test 3 โ Multi-turn code edit, same snippet + instruction (full-file rewrite asked)
- ๐ ๏ธ Test 4 โ Agentic
<nx_tool>file ops + 4000px vision ID - ๐ฎ Test 5 โ Why no 27Bโ32B dense on 6 GB VRAM (measured ceiling, labeled projection)
- ๐ป Quickstart โ
llama-server(CUDA) - Quickstart โ Ollama
Modelfile - ๐ฆ Files
- ๐ก๏ธ Limitations
- License & Non-Commercial Restriction (CC-BY-NC-SA-4.0)
- Citation
- ๐๏ธ Custom Architecture & Multimodal Upgrade
๐ NYXIS-30B-A3B โ Qwen3-VL MoE + Native Vision for the NYXIS Local Harness
A 30.5B-total / 3.3B-active sparse Mixture-of-Experts Vision-Language model
(Qwen3-VL-30B-A3B-Instruct lineage) packaged for the NYXIS desktop harness:
a P-core-pinned llama-server backend, a custom 51-trajectory agentic
SFT/QLoRA distillation for <nx_tool> workspace file ops, live unified
+X / -Y diffs, single-pass HTML/CSS/JS app synthesis, and visual entity
verification. This card documents the exact build that runs on a 6 GB
laptop GPU at 2,663 MB idle / 2,735 MB active dedicated VRAM.
๐๏ธ Custom Architecture & Multimodal Upgrade
The unified nyxis-vl-30b-a3b checkpoint pairs the Qwen3-VL-30B-A3B
Vision-Language MoE (128 experts per layer, top-8 active, 3.3B active
params per token) with a GPU-resident vision projector:
- Text tower:
nyxis-vl-30b-a3b-Q2_K.gguf(11.3 GB) โ Q2_K quant of the 30B MoE. Attention and all dense layers stay on GPU; 45 of 48 MoE layers ride CPU (--n-cpu-moe 45), 3 stay on GPU for prefill/decode speed. - Vision projector:
mmproj-Qwen3-VL-30B-Q8_0.gguf(712 MB, officialggml-orgQ8_0 quant) loaded with--mmproj-device CUDA0, so the ViT encode runs on GPU (~10x faster than the old CPU-encode path). - KV cache:
q4_0for K and V (-ctk q4_0 -ctv q4_0) โ measured zero decode loss vs F16 cache, ~200 MB VRAM saved. - Agentic distillation: 51 gold-standard trajectories
(
<nx_tool>file ops, unified diffs, single-file app synthesis) distilled into the harness prompts, plus a forced-JSON tool backstop: any file-op ask that decodes without a runnable<nx_tool>tag is re-run as a grammar-constrained{"op","path","content","find","replace"}pass, so roleplayed tool calls are impossible. - Visual entity verification: identification turns carry a character signature sheet; the model must match pixels to a sheet line confidently or describe visuals only with an explicit can't-confirm close.
โก Why It Runs on a 6 GB RTX 4050 Laptop (2.73 GB VRAM)
Hugging Face's default VRAM badge assumes full GPU offload. NYXIS does the opposite โ an asymmetric GPU/CPU MoE split:
| Lever | Setting | Effect |
|---|---|---|
| MoE split | --n-cpu-moe 45 |
45 expert layers on CPU, 3 on GPU |
| KV cache | -ctk q4_0 -ctv q4_0 |
~200 MB saved, no decode loss |
| Vision projector | Q8_0 mmproj on CUDA0 |
712 MB vs 1.08 GB F16, GPU encode |
| Threads | fixed -t 4 -tb 4 (b95; measured 21 tok/s vs 16 on -t 6) |
peak CPU-MoE decode, no oversubscription |
| ๐ก๏ธ RAM policy | Bounded 6.0 GB working-set ceiling (b97) โ hot MoE trunk + KV stay resident, cold --mmap pages trim to standby (game-available) |
engine disk I/O 0.00 bytes/s idle; GGUF read once at boot |
Measured dedicated engine VRAM (per-process GPU Process Memory, not the
shared WDDM board total): 2,663 MB idle / 2,716 MB active (b97) โ strictly
under the 3.0 GB cap, so a 30B Vision+Code model runs at full speed while
games like Minecraft stay open.
b97 RAM note (measured A/B, same box/prompts): the engine's hot decode
set is ~5.5 GB, so the b97 working-set ceiling is 6.0 GB (hardWS
min 1.5 GB / max 6.0 GB + stray-Ollama-runner eviction + -b 512 batch
buffers + 2 h idle hold for 1.3 s warm greetings). A 3.4 GB ceiling was
tested and rejected: decode fell to 12.2 tok/s with 3.44M page faults per
120 tokens, while freeing zero game RAM (evicted pages sit in standby,
which Windows hands to games anyway). At 6.0 GB: working set 5,722 MB
steady, text decode 20.1 tok/s, vision decode 14.4โ20.0 tok/s, warm "Hi"
1.36 s, committed RAM 64.7%, engine disk I/O 0.00 bytes/s idle.
๐ Real-World Hardware Benchmarks (RTX 4050 6 GB + 16 GB RAM, i5-13420H)
| Metric | Measured |
|---|---|
| Dedicated GPU VRAM | 2,663 MB idle / 2,735 MB active (< 3.0 GB cap) |
| ๐ง Engine CPU working set (b97 6 GB cap) | 5,722 MB steady (โค 6.0 GB ceiling; measured ~5.5 GB hot decode set) |
| Warm chat TTFT / greeting turn | 424 ms TTFT / 1.3โ2.9 s total (b85โb94 box range) |
| Code-gen decode (no-spec default) | 19.5โ20.6 tok/s (b95 vision-turn decode: 20.0 tok/s @ -t 4) |
Code-edit decode (ngram-mod 12/3 opt-in) |
2.35x speedup, 48.4 tok/s (b77 direct-server; NOT reproduced in b96 harness A/B โ see below, default stays none) |
| โ๏ธ Visual entity ID, 4000px reference test photo | 100% (verified every build incl. b95โb97 + suite) |
| Workspace file-op + diff accuracy | 100% (50/50 automated test suite) |
| Cold first-look vision (full ID sheet) | ~20โ35 s (prefill-bound on CPU MoE; 6.8 s direct-probe floor with minimal prompt) |
Notes: ngram-mod is an opt-in (NYXIS_SPEC_TYPE=ngram-mod) because it
slows long vision prompts even as it accelerates code edits. Vision
first-look is prefill-bound (character sheet + visual tokens through CPU
MoE layers); short-prompt probes hit 6.8 s on the same weights.
โ๏ธ Empirical Head-to-Head Benchmarks (Measured on RTX 4050 6 GB + i5-13420H, 16 GB RAM)
b96 battery (core/bench_b96.cjs + core/bench_b96b.cjs, identical prompts
per test, all numbers measured โ zero fabricated). Contenders on the same box:
A = nyxis-vl-30b-a3b + NYXIS harness (this release, default none);
B = stock text-only nyxis-30b-abliterated Q2_K MoE (10.7 GB, via Ollama);
C = nyxis-student 4B Q4 dense (via Ollama). No 27Bโ32B dense GGUF is on
disk and this link (~0.5โ2.6 MB/s) rules out a 10 GB download, so dense-32B
is a labeled projection from measured bandwidth (see Test 5).
๐งช Test 1 โ VRAM (per-process dedicated) & CPU working set
| Model | Idle VRAM | Load VRAM | CPU working set under load |
|---|---|---|---|
| A: NYXIS-VL-30B-A3B + harness | 2,663 MB | 2,716 MB (b97) | 5,722 MB under 6.0 GB hardWS ceiling (b97: engine disk I/O 0.00 bytes/s idle) |
B: stock 30B-A3B-Q2_K (Ollama, num_gpu 20) |
n/a (Ollama loads on demand) | 4,803 MB | 7.2 GB |
| C: 4B Q4 dense student (Ollama, full offload) | n/a (loads on demand) | 3,251 MB | runner ~0.8 GB |
Stock Ollama defaults fail to load B at all: llama-server reported out-of-memory during startup: CUDA error: out of memory โ even with the
NYXIS engine unloaded. B only runs with a manual partial offload
(num_gpu 20, ~20/48 layers on GPU). This is the 6 GB VRAM wall this
release is built to fit under.
๐ฅ Test 2 โ Cold & warm code generation (identical 400-token HTML/JS todo-app prompt)
| Model | Prefill | Decode | Wall (400-token cap where applied) |
|---|---|---|---|
| A: NYXIS-VL-30B-A3B + harness | 724 tok @ 99.9 tok/s (prefix-cached) | 1,786 tok @ 21.0 tok/s (cap ignored, full app emitted) | 94.3 s |
A + ngram-mod 12/3 |
4,005 tok @ 118.1 tok/s (cold engine) | 1,786 tok @ 15.4 tok/s | 151.5 s |
B: stock 30B MoE (num_gpu 20) |
55 tok @ 72.8 tok/s | 400 tok @ 25.9 tok/s | 29.7 s |
| C: 4B dense | 55 tok @ 280.9 tok/s | 400 tok @ 49.6 tok/s | 14.7 s |
๐ง Test 3 โ Multi-turn code edit, same snippet + instruction (full-file rewrite asked)
| Model | Prefill | Decode | Result |
|---|---|---|---|
| A: NYXIS-VL-30B-A3B + harness | 2,829 tok @ 154.9 tok/s (cached context) | 26 tok @ 19.0 tok/s | self-truncated at 26 tok (incomplete file); re-run: complete 307-char file in 26.0 s โ model-side EOS variance, 1/2 complete |
A + ngram-mod 12/3 |
2,829 tok @ 109.3 tok/s | 26 tok @ 14.0 tok/s | same truncation; the b77 2.35x edit win did not reproduce in this build (codegen 15.4 vs 21.0, edit 14.0 vs 19.0) โ default stays none |
| B: stock 30B MoE | 116 tok @ 187.2 tok/s | 400 tok @ 19.0 tok/s | complete file, 21.8 s |
| C: 4B dense | 163 tok @ 269.6 tok/s | 246 tok @ 20.2 tok/s | 18.6 s |
๐ ๏ธ Test 4 โ Agentic <nx_tool> file ops + 4000px vision ID
Natural prompt: "write workspace file b96_live.txt with hello world, then edit it to HELLO WORLD". Raw models get the same prompt with no harness.
| Model | Write hello world |
Chained edit to HELLO WORLD |
Vision (4000px reference test photo) |
|---|---|---|---|
| A: NYXIS-VL-30B-A3B + harness | 4/4 (wrote N bytes (+1 -0) receipts) |
2/4 (1 scripted + 1 ngram run landed; 2 runs left lowercase on disk while claiming success โ chained-edit flakiness is real, write path is solid) | correct ID every run (once hedged "consistent with", rest confident); 6.8โ16.8 s |
| B: stock 30B MoE | 0% โ emits <think> deliberation, no runnable tag |
0% | structural fail: HTTP 400 Multimodal data provided, but model does not support multimodal requests |
| C: 4B dense | 0% โ writes a bash echo > tutorial instead |
0% | no image input support (structural fail) |
Takeaway: without the harness (forced-JSON tool backstop + receipts), 0% of natural file-op prompts execute on any raw model โ they narrate instead. With it, writes are 100% and chained edits ~50% on this build.
๐ฎ Test 5 โ Why no 27Bโ32B dense on 6 GB VRAM (measured ceiling, labeled projection)
- Fit (measured): the 10.7 GB 30B MoE OOMs at full offload; only a
manual
num_gpu 20split (4.8 GB VRAM + 7.2 GB RAM) runs. A dense 32B Q2_K (~11โ12 GB) has no MoE layers to leave on CPU cheaply โ every token touches every weight โ so it faces the same OOM with no equivalent split. - Speed (projected from measured bandwidth): A's decode (21 tok/s) streams
3.3B active params (1.2 GB/token at Q2_K) through the CPU-RAM path. A dense 32B streams10x the bytes per token through the same DDR5 bus โ **2โ3 tok/s projected decode**, and prefill (11+ GB per pass) lands in the tens of seconds per turn โ before KV cache, which doesn't fit either. - Verdict: on this 6 GB box, dense-32B-class is OOM-or-unusable; the 3.3B-active MoE split is what makes 30B-class weights interactive here.
๐ป Quickstart โ llama-server (CUDA)
llama-server \
-m nyxis-vl-30b-a3b-Q2_K.gguf \
--mmproj mmproj-Qwen3-VL-30B-Q8_0.gguf --mmproj-device CUDA0 \
--n-cpu-moe 45 -c 4096 -ctk q4_0 -ctv q4_0 \
-t 4 -tb 4 -b 512 -ub 256 -fa on \
--host 127.0.0.1 --port 11435 --no-webui
# Edit-session speculation booster (opt-in):
# --spec-type ngram-mod --spec-draft-n-max 12 --spec-draft-n-min 3
Vision request shape (OpenAI-compat /v1/chat/completions):
{"model": "nyxis-vl-30b", "messages": [
{"role": "system", "content": "<vision_sys + ID-REF sheet>"},
{"role": "user", "content": [
{"type": "text", "text": "Who is this?"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]}],
"max_tokens": 120, "temperature": 0.55, "cache_prompt": true}
Photo turns are resized to 288px longest side before sending (detail / screenshot turns keep 448px); non-code vision turns cap at 120 tokens.
Quickstart โ Ollama Modelfile
See Modelfile in this repo (30B VL profile: num_ctx 6144,
num_predict 800, temperature 0.55):
ollama create nyxis-vl-30b -f Modelfile
ollama run nyxis-vl-30b
๐ฆ Files
| File | Size | Notes |
|---|---|---|
nyxis-vl-30b-a3b-Q2_K.gguf |
~11.3 GB | MoE text tower (Q2_K) |
mmproj-Qwen3-VL-30B-Q8_0.gguf |
712 MB | Official Q8_0 vision projector |
README.md |
โ | This card |
Modelfile |
โ | Ollama profile |
๐ก๏ธ Limitations
- Cold vision first-look carries the full ID sheet in-context (16.8 s on this laptop class as of b95; repeat turns reuse the cached prefix and are far faster).
- Draft-model speculative decoding is unsupported for this pair (target and
0.6B draft vocabs are incompatible);
ngram-modis the supported booster. - Q8_0 projector tensors require a llama.cpp build with Q8 mtmd support (b12387+ class); hand-rolled Q8 quants of the F16 projector fail to load.
License & Non-Commercial Restriction (CC-BY-NC-SA-4.0)
This model, its custom agentic distillation trajectories, and the NYXIS harness configurations are licensed strictly under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC-BY-NC-SA-4.0). Commercial use, cloud inference provider deployment, paid API hosting, and enterprise/data-center usage are strictly prohibited. Built exclusively for local personal and research use on consumer hardware.
Citation
Base weights: Qwen/Qwen3-VL-30B-A3B-Instruct (Apache-2.0, upstream only).
This release โ Q2_K quant, Q8_0 vision projector pairing, agentic
distillation trajectories, harness configurations, and this card โ is
licensed CC-BY-NC-SA-4.0 (see above) by the NYXIS project.
- Downloads last month
- 341
2-bit
Model tree for NoobieToPro/NYXIS-30B-A3B-GGUF
Base model
Qwen/Qwen3-VL-30B-A3B-Instruct