mlx-community/RakutenAI-3.0-MLX-4bit

This model was converted to MLX format from Rakuten/RakutenAI-3.0 using mlx-lm version 0.31.0. Refer to the original model card for more details on the model.

Convert:❯ mlx_lm.convert --hf-path ./RakutenAI-3.0 --mlx-path ./lm-studio/models/limegreenpeper1/rakutenai3-mlx-4bit -q --q-bits 4 --trust-remote-code

[INFO] Loading rope_parameters's factor field must be a float >= 1, got 40 rope_parameters's beta_fast field must be a float, got 32 rope_parameters's beta_slow field must be a float, got 1 [INFO] Using dtype: bfloat16 [INFO] Quantizing [INFO] Quantized model with 4.502 bits per weight.

Inference: 20.06 tok/sec M3 Ultra (CTX 128K - LM Studio MLX v1.3.0)