gpt-oss-20b-Autoround-a16w4: Compiled for SiMa.ai Modalix

Overview

Compiled GPT-OSS 20B mixture-of-experts model for SiMa.ai Modalix.

• Architecture: 24 transformer layers, 32 experts per layer, four selected experts per token.
• Language weights: AutoRound, symmetric INT4, block size 256.
• Routers: RTN, symmetric INT8 per output channel.
• Output head: RTN, symmetric INT4, block size 256.
• Activations: BF16; embeddings and KV cache quantized to INT8.
• Maximum context length: 4,096 tokens, including prompt and generated tokens.
• Prefill group size: 128; sliding-attention window: 128 tokens.
• Filter sharing: enabled by default for language components.
• Model SDK: 3.0.0.

The encoded quantized weights and mixed-precision exceptions were preserved from the local prequantized checkpoint. The saved recipe.yaml and compilation.json record its recipe, source checksums and compiler revisions. The original BF16 source was unsloth/gpt-oss-20b-BF16.

Performance

To be added.

Download and run

Use a Modalix LLiMa runtime supporting GPT-OSS/MoE and the compiled configuration. Download the unpacked model directly on the device:

hf download florianvoss/gpt-oss-20b-Autoround-a16w4 \
    --local-dir /media/nvme/llima/models/gpt-oss-20b-Autoround-a16w4
llima run /media/nvme/llima/models/gpt-oss-20b-Autoround-a16w4

The package contains devkit/ with runtime configuration, tokenizer, embeddings and attention sinks, and elf_files/ with all 1,683 compiled executables. This compiled package cannot be loaded with Transformers.

Recompilation

Activate a matching Model Compiler environment and provide the same prequantized checkpoint. That checkpoint is local-only and is not available through a public Hugging Face download at publication time.

MODEL_DIR=/path/to/gpt-oss-20b-Autoround-Safetensors bash compile.sh
llima-deploy CompiledModels/gpt-oss-20b-Autoround-a16w4 gpt-oss-20b-Autoround-a16w4-deployed

The included compile.sh defaults to 4K context and six parallel compiler jobs. Embedding and KV-cache quantization and language filter sharing use the compiler defaults. Additional flags are forwarded to llima-compile. The published build used 16 compiler processes with one simulator thread each, from LLiMa develop revision b972c9ac35f93143a2baf88154454988074ca8d7.

Validation

All 1,683 components compiled successfully with exit code 0. The model was unpacked using llima-deploy; the ELF inventory and runtime configuration were checked, and uploaded files were verified by checksum.

Modalix generation, accuracy and performance testing have not been performed for this package. Compilation does not establish runtime output quality. The immutable original BF16 revision was not recorded; exact prequantized source file checksums are included in compilation.json.

License

Apache 2.0. The upstream LICENSE and USAGE_POLICY are included. License-file provenance is recorded in compilation.json.