๐Ÿš€ NYXIS-30B-A3B โ€” Qwen3-VL MoE + Native Vision for the NYXIS Local Harness

A 30.5B-total / 3.3B-active sparse Mixture-of-Experts Vision-Language model (Qwen3-VL-30B-A3B-Instruct lineage) packaged for the NYXIS desktop harness: a P-core-pinned llama-server backend, a custom 51-trajectory agentic SFT/QLoRA distillation for <nx_tool> workspace file ops, live unified +X / -Y diffs, single-pass HTML/CSS/JS app synthesis, and visual entity verification. This card documents the exact build that runs on a 6 GB laptop GPU at 2,663 MB idle / 2,735 MB active dedicated VRAM.

๐Ÿ—๏ธ Custom Architecture & Multimodal Upgrade

The unified nyxis-vl-30b-a3b checkpoint pairs the Qwen3-VL-30B-A3B Vision-Language MoE (128 experts per layer, top-8 active, 3.3B active params per token) with a GPU-resident vision projector:

  • Text tower: nyxis-vl-30b-a3b-Q2_K.gguf (11.3 GB) โ€” Q2_K quant of the 30B MoE. Attention and all dense layers stay on GPU; 45 of 48 MoE layers ride CPU (--n-cpu-moe 45), 3 stay on GPU for prefill/decode speed.
  • Vision projector: mmproj-Qwen3-VL-30B-Q8_0.gguf (712 MB, official ggml-org Q8_0 quant) loaded with --mmproj-device CUDA0, so the ViT encode runs on GPU (~10x faster than the old CPU-encode path).
  • KV cache: q4_0 for K and V (-ctk q4_0 -ctv q4_0) โ€” measured zero decode loss vs F16 cache, ~200 MB VRAM saved.
  • Agentic distillation: 51 gold-standard trajectories (<nx_tool> file ops, unified diffs, single-file app synthesis) distilled into the harness prompts, plus a forced-JSON tool backstop: any file-op ask that decodes without a runnable <nx_tool> tag is re-run as a grammar-constrained {"op","path","content","find","replace"} pass, so roleplayed tool calls are impossible.
  • Visual entity verification: identification turns carry a character signature sheet; the model must match pixels to a sheet line confidently or describe visuals only with an explicit can't-confirm close.

โšก Why It Runs on a 6 GB RTX 4050 Laptop (2.73 GB VRAM)

Hugging Face's default VRAM badge assumes full GPU offload. NYXIS does the opposite โ€” an asymmetric GPU/CPU MoE split:

Lever Setting Effect
MoE split --n-cpu-moe 45 45 expert layers on CPU, 3 on GPU
KV cache -ctk q4_0 -ctv q4_0 ~200 MB saved, no decode loss
Vision projector Q8_0 mmproj on CUDA0 712 MB vs 1.08 GB F16, GPU encode
Threads fixed -t 4 -tb 4 (b95; measured 21 tok/s vs 16 on -t 6) peak CPU-MoE decode, no oversubscription
๐Ÿ›ก๏ธ RAM policy Bounded 6.0 GB working-set ceiling (b97) โ€” hot MoE trunk + KV stay resident, cold --mmap pages trim to standby (game-available) engine disk I/O 0.00 bytes/s idle; GGUF read once at boot

Measured dedicated engine VRAM (per-process GPU Process Memory, not the shared WDDM board total): 2,663 MB idle / 2,716 MB active (b97) โ€” strictly under the 3.0 GB cap, so a 30B Vision+Code model runs at full speed while games like Minecraft stay open.

b97 RAM note (measured A/B, same box/prompts): the engine's hot decode set is ~5.5 GB, so the b97 working-set ceiling is 6.0 GB (hardWS min 1.5 GB / max 6.0 GB + stray-Ollama-runner eviction + -b 512 batch buffers + 2 h idle hold for 1.3 s warm greetings). A 3.4 GB ceiling was tested and rejected: decode fell to 12.2 tok/s with 3.44M page faults per 120 tokens, while freeing zero game RAM (evicted pages sit in standby, which Windows hands to games anyway). At 6.0 GB: working set 5,722 MB steady, text decode 20.1 tok/s, vision decode 14.4โ€“20.0 tok/s, warm "Hi" 1.36 s, committed RAM 64.7%, engine disk I/O 0.00 bytes/s idle.

๐Ÿ“Š Real-World Hardware Benchmarks (RTX 4050 6 GB + 16 GB RAM, i5-13420H)

Metric Measured
Dedicated GPU VRAM 2,663 MB idle / 2,735 MB active (< 3.0 GB cap)
๐Ÿ”ง Engine CPU working set (b97 6 GB cap) 5,722 MB steady (โ‰ค 6.0 GB ceiling; measured ~5.5 GB hot decode set)
Warm chat TTFT / greeting turn 424 ms TTFT / 1.3โ€“2.9 s total (b85โ€“b94 box range)
Code-gen decode (no-spec default) 19.5โ€“20.6 tok/s (b95 vision-turn decode: 20.0 tok/s @ -t 4)
Code-edit decode (ngram-mod 12/3 opt-in) 2.35x speedup, 48.4 tok/s (b77 direct-server; NOT reproduced in b96 harness A/B โ€” see below, default stays none)
โš”๏ธ Visual entity ID, 4000px reference test photo 100% (verified every build incl. b95โ€“b97 + suite)
Workspace file-op + diff accuracy 100% (50/50 automated test suite)
Cold first-look vision (full ID sheet) ~20โ€“35 s (prefill-bound on CPU MoE; 6.8 s direct-probe floor with minimal prompt)

Notes: ngram-mod is an opt-in (NYXIS_SPEC_TYPE=ngram-mod) because it slows long vision prompts even as it accelerates code edits. Vision first-look is prefill-bound (character sheet + visual tokens through CPU MoE layers); short-prompt probes hit 6.8 s on the same weights.

โš”๏ธ Empirical Head-to-Head Benchmarks (Measured on RTX 4050 6 GB + i5-13420H, 16 GB RAM)

b96 battery (core/bench_b96.cjs + core/bench_b96b.cjs, identical prompts per test, all numbers measured โ€” zero fabricated). Contenders on the same box: A = nyxis-vl-30b-a3b + NYXIS harness (this release, default none); B = stock text-only nyxis-30b-abliterated Q2_K MoE (10.7 GB, via Ollama); C = nyxis-student 4B Q4 dense (via Ollama). No 27Bโ€“32B dense GGUF is on disk and this link (~0.5โ€“2.6 MB/s) rules out a 10 GB download, so dense-32B is a labeled projection from measured bandwidth (see Test 5).

๐Ÿงช Test 1 โ€” VRAM (per-process dedicated) & CPU working set

Model Idle VRAM Load VRAM CPU working set under load
A: NYXIS-VL-30B-A3B + harness 2,663 MB 2,716 MB (b97) 5,722 MB under 6.0 GB hardWS ceiling (b97: engine disk I/O 0.00 bytes/s idle)
B: stock 30B-A3B-Q2_K (Ollama, num_gpu 20) n/a (Ollama loads on demand) 4,803 MB 7.2 GB
C: 4B Q4 dense student (Ollama, full offload) n/a (loads on demand) 3,251 MB runner ~0.8 GB

Stock Ollama defaults fail to load B at all: llama-server reported out-of-memory during startup: CUDA error: out of memory โ€” even with the NYXIS engine unloaded. B only runs with a manual partial offload (num_gpu 20, ~20/48 layers on GPU). This is the 6 GB VRAM wall this release is built to fit under.

๐Ÿ”ฅ Test 2 โ€” Cold & warm code generation (identical 400-token HTML/JS todo-app prompt)

Model Prefill Decode Wall (400-token cap where applied)
A: NYXIS-VL-30B-A3B + harness 724 tok @ 99.9 tok/s (prefix-cached) 1,786 tok @ 21.0 tok/s (cap ignored, full app emitted) 94.3 s
A + ngram-mod 12/3 4,005 tok @ 118.1 tok/s (cold engine) 1,786 tok @ 15.4 tok/s 151.5 s
B: stock 30B MoE (num_gpu 20) 55 tok @ 72.8 tok/s 400 tok @ 25.9 tok/s 29.7 s
C: 4B dense 55 tok @ 280.9 tok/s 400 tok @ 49.6 tok/s 14.7 s

๐Ÿ”ง Test 3 โ€” Multi-turn code edit, same snippet + instruction (full-file rewrite asked)

Model Prefill Decode Result
A: NYXIS-VL-30B-A3B + harness 2,829 tok @ 154.9 tok/s (cached context) 26 tok @ 19.0 tok/s self-truncated at 26 tok (incomplete file); re-run: complete 307-char file in 26.0 s โ€” model-side EOS variance, 1/2 complete
A + ngram-mod 12/3 2,829 tok @ 109.3 tok/s 26 tok @ 14.0 tok/s same truncation; the b77 2.35x edit win did not reproduce in this build (codegen 15.4 vs 21.0, edit 14.0 vs 19.0) โ€” default stays none
B: stock 30B MoE 116 tok @ 187.2 tok/s 400 tok @ 19.0 tok/s complete file, 21.8 s
C: 4B dense 163 tok @ 269.6 tok/s 246 tok @ 20.2 tok/s 18.6 s

๐Ÿ› ๏ธ Test 4 โ€” Agentic <nx_tool> file ops + 4000px vision ID

Natural prompt: "write workspace file b96_live.txt with hello world, then edit it to HELLO WORLD". Raw models get the same prompt with no harness.

Model Write hello world Chained edit to HELLO WORLD Vision (4000px reference test photo)
A: NYXIS-VL-30B-A3B + harness 4/4 (wrote N bytes (+1 -0) receipts) 2/4 (1 scripted + 1 ngram run landed; 2 runs left lowercase on disk while claiming success โ€” chained-edit flakiness is real, write path is solid) correct ID every run (once hedged "consistent with", rest confident); 6.8โ€“16.8 s
B: stock 30B MoE 0% โ€” emits <think> deliberation, no runnable tag 0% structural fail: HTTP 400 Multimodal data provided, but model does not support multimodal requests
C: 4B dense 0% โ€” writes a bash echo > tutorial instead 0% no image input support (structural fail)

Takeaway: without the harness (forced-JSON tool backstop + receipts), 0% of natural file-op prompts execute on any raw model โ€” they narrate instead. With it, writes are 100% and chained edits ~50% on this build.

๐ŸŽฎ Test 5 โ€” Why no 27Bโ€“32B dense on 6 GB VRAM (measured ceiling, labeled projection)

  • Fit (measured): the 10.7 GB 30B MoE OOMs at full offload; only a manual num_gpu 20 split (4.8 GB VRAM + 7.2 GB RAM) runs. A dense 32B Q2_K (~11โ€“12 GB) has no MoE layers to leave on CPU cheaply โ€” every token touches every weight โ€” so it faces the same OOM with no equivalent split.
  • Speed (projected from measured bandwidth): A's decode (21 tok/s) streams 3.3B active params (1.2 GB/token at Q2_K) through the CPU-RAM path. A dense 32B streams 10x the bytes per token through the same DDR5 bus โ†’ **2โ€“3 tok/s projected decode**, and prefill (11+ GB per pass) lands in the tens of seconds per turn โ€” before KV cache, which doesn't fit either.
  • Verdict: on this 6 GB box, dense-32B-class is OOM-or-unusable; the 3.3B-active MoE split is what makes 30B-class weights interactive here.

๐Ÿ’ป Quickstart โ€” llama-server (CUDA)

llama-server \
  -m nyxis-vl-30b-a3b-Q2_K.gguf \
  --mmproj mmproj-Qwen3-VL-30B-Q8_0.gguf --mmproj-device CUDA0 \
  --n-cpu-moe 45 -c 4096 -ctk q4_0 -ctv q4_0 \
  -t 4 -tb 4 -b 512 -ub 256 -fa on \
  --host 127.0.0.1 --port 11435 --no-webui
# Edit-session speculation booster (opt-in):
#   --spec-type ngram-mod --spec-draft-n-max 12 --spec-draft-n-min 3

Vision request shape (OpenAI-compat /v1/chat/completions):

{"model": "nyxis-vl-30b", "messages": [
  {"role": "system", "content": "<vision_sys + ID-REF sheet>"},
  {"role": "user", "content": [
    {"type": "text", "text": "Who is this?"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]}],
 "max_tokens": 120, "temperature": 0.55, "cache_prompt": true}

Photo turns are resized to 288px longest side before sending (detail / screenshot turns keep 448px); non-code vision turns cap at 120 tokens.

Quickstart โ€” Ollama Modelfile

See Modelfile in this repo (30B VL profile: num_ctx 6144, num_predict 800, temperature 0.55):

ollama create nyxis-vl-30b -f Modelfile
ollama run nyxis-vl-30b

๐Ÿ“ฆ Files

File Size Notes
nyxis-vl-30b-a3b-Q2_K.gguf ~11.3 GB MoE text tower (Q2_K)
mmproj-Qwen3-VL-30B-Q8_0.gguf 712 MB Official Q8_0 vision projector
README.md โ€” This card
Modelfile โ€” Ollama profile

๐Ÿ›ก๏ธ Limitations

  • Cold vision first-look carries the full ID sheet in-context (16.8 s on this laptop class as of b95; repeat turns reuse the cached prefix and are far faster).
  • Draft-model speculative decoding is unsupported for this pair (target and 0.6B draft vocabs are incompatible); ngram-mod is the supported booster.
  • Q8_0 projector tensors require a llama.cpp build with Q8 mtmd support (b12387+ class); hand-rolled Q8 quants of the F16 projector fail to load.

License & Non-Commercial Restriction (CC-BY-NC-SA-4.0)

This model, its custom agentic distillation trajectories, and the NYXIS harness configurations are licensed strictly under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC-BY-NC-SA-4.0). Commercial use, cloud inference provider deployment, paid API hosting, and enterprise/data-center usage are strictly prohibited. Built exclusively for local personal and research use on consumer hardware.

Citation

Base weights: Qwen/Qwen3-VL-30B-A3B-Instruct (Apache-2.0, upstream only). This release โ€” Q2_K quant, Q8_0 vision projector pairing, agentic distillation trajectories, harness configurations, and this card โ€” is licensed CC-BY-NC-SA-4.0 (see above) by the NYXIS project.

Downloads last month
341
GGUF
Model size
31B params
Architecture
qwen3vlmoe
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for NoobieToPro/NYXIS-30B-A3B-GGUF

Quantized
(61)
this model