shisa-ai/shisa-de-2-FP8-dynamic

Shisa DE-2 FP8-Dynamic

An FP8-dynamic quantization of shisa-ai/shisa-de-2, at 27.2 GB against 49 GB for the original. The readout contract, inference policy, usage and benchmarks are the same as the original model's; see its card.

Quantization

Scheme	FP8_DYNAMIC: 8-bit float weights with per-channel scales, dynamic per-token activation scales
Method	Data-free, with llm-compressor 0.13.0 model_free_ptq
Quantized	The Linear layers inside the transformer blocks
Kept in BF16	Vision tower, audio tower, token embeddings, output head, MoE router, normalization layers

recipe.yaml has the equivalent recipe.

Readout test

The 231 public JevBench items, read through the Shisa DE SDK from vLLM 0.30.0 on one RTX PRO 6000, raw probabilities, against the original model on the same server setup:

	Original (BF16)	FP8-dynamic
Single read, easy / medium / hard	48 / 72 / 83	48 / 72 / 84
Adaptive policy, easy / medium / hard	48 / 72 / 86	48 / 72 / 85
Same answer as the original, single read		224 of 231
Same answer as the original, adaptive policy		224 of 231
Mean change in top probability, single read		0.026

Serving

vllm serve shisa-ai/shisa-de-2-FP8-dynamic

from shisa_de import DecisionModel

de = DecisionModel.from_endpoint(
    "http://127.0.0.1:8000/v1",
    model="shisa-ai/shisa-de-2-FP8-dynamic",
    tokenizer="shisa-ai/shisa-de-2",   # also applies the original model's calibration
    api_key="",
)
de.classify({"ticket": "I was charged twice."}, {"intent": ["refund", "cancel"]})