Instructions to use ggml-org/Laya-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ggml-org/Laya-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ggml-org/Laya-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf ggml-org/Laya-GGUF:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ggml-org/Laya-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf ggml-org/Laya-GGUF:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ggml-org/Laya-GGUF:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf ggml-org/Laya-GGUF:BF16
Use Docker
docker model run hf.co/ggml-org/Laya-GGUF:BF16
- Llama new
- LM Studio
- Jan
- Ollama
How to use ggml-org/Laya-GGUF with Ollama:
ollama run hf.co/ggml-org/Laya-GGUF:BF16
- Unsloth Desktop
- Docker Model Runner
How to use ggml-org/Laya-GGUF with Docker Model Runner:
docker model run hf.co/ggml-org/Laya-GGUF:BF16
- Lemonade
How to use ggml-org/Laya-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ggml-org/Laya-GGUF:BF16
Run and chat with the model
lemonade run user.Laya-GGUF-BF16
List all available models
lemonade list
- Atomic Chat
Download convert.log from ggml-org/Laya-GGUF: direct link, hf CLI and curl.
- Browser
- Download file 56 kB
-
https://huggingface.co/ggml-org/Laya-GGUF/resolve/main/convert.log
- Command line
-
hf download hf://ggml-org/Laya-GGUF/convert.log
-
curl -L -o convert.log https://huggingface.co/ggml-org/Laya-GGUF/resolve/main/convert.log
56 kB
| + OUTPUT_DIR=./upload-Laya | |
| + LLAMA_CPP=llama.cpp | |
| + DISPLAY_NAME=Laya | |
| + QUANTIZE=llama.cpp/build/bin/llama-quantize | |
| + python3 llama.cpp/convert_hf_to_gguf.py ./model-temp-Laya-PRIMARY --outtype bf16 --outfile ./upload-Laya/Laya-BF16.gguf --model-name Laya | |
| INFO:hf-to-gguf:Loading model: model-temp-Laya-PRIMARY | |
| INFO:hf-to-gguf:gguf: detected ModernBert decision checkpoint | |
| INFO:hf-to-gguf:Model architecture: ModernBertDecisionModel | |
| INFO:hf-to-gguf:gguf: detected ModernBert decision checkpoint | |
| INFO:hf-to-gguf:gguf: indexing model part 'model.safetensors' | |
| INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only | |
| INFO:hf-to-gguf:Exporting model... | |
| INFO:hf-to-gguf:token_embd_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:token_embd.weight, torch.float16 --> BF16, shape = {1024, 50368} | |
| INFO:hf-to-gguf:output_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.0.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.0.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.0.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.0.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.0.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.1.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.1.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.1.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.1.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.1.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.1.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.10.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.10.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.10.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.10.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.10.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.10.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.11.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.11.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.11.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.11.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.11.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.11.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.12.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.12.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.12.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.12.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.12.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.12.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.13.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.13.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.13.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.13.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.13.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.13.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.14.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.14.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.14.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.14.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.14.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.14.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.15.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.15.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.15.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.15.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.15.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.15.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.16.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.16.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.16.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.16.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.16.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.16.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.17.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.17.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.17.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.17.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.17.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.17.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.18.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.18.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.18.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.18.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.18.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.18.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.19.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.19.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.19.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.19.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.19.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.19.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.2.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.2.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.2.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.2.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.2.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.2.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.20.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.20.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.20.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.20.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.20.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.20.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.21.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.21.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.21.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.21.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.21.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.21.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.22.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.22.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.22.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.22.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.22.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.22.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.23.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.23.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.23.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.23.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.23.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.23.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.24.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.24.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.24.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.24.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.24.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.24.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.25.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.25.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.25.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.25.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.25.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.25.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.26.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.26.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.26.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.26.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.26.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.26.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.27.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.27.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.27.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.27.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.27.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.27.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.3.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.3.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.3.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.3.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.3.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.3.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.4.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.4.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.4.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.4.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.4.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.4.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.5.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.5.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.5.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.5.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.5.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.5.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.6.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.6.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.6.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.6.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.6.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.6.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.7.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.7.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.7.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.7.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.7.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.7.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.8.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.8.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.8.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.8.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.8.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.8.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.9.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.9.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.9.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.9.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 5248} | |
| INFO:hf-to-gguf:blk.9.ffn_down.weight, torch.float16 --> BF16, shape = {2624, 1024} | |
| INFO:hf-to-gguf:blk.9.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.ffn_up.bias, torch.float16 --> F32, shape = {4096} | |
| INFO:hf-to-gguf:blk.28.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 4096} | |
| INFO:hf-to-gguf:blk.28.ffn_down.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.ffn_down.weight, torch.float16 --> BF16, shape = {4096, 1024} | |
| INFO:hf-to-gguf:blk.28.attn_norm.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.ffn_norm.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.attn_qkv.bias, torch.float16 --> F32, shape = {3072} | |
| INFO:hf-to-gguf:blk.28.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.28.attn_output.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.28.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:blk.29.ffn_up.bias, torch.float16 --> F32, shape = {4096} | |
| INFO:hf-to-gguf:blk.29.ffn_up.weight, torch.float16 --> BF16, shape = {1024, 4096} | |
| INFO:hf-to-gguf:blk.29.ffn_down.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.ffn_down.weight, torch.float16 --> BF16, shape = {4096, 1024} | |
| INFO:hf-to-gguf:blk.29.attn_norm.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.attn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.ffn_norm.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.ffn_norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.attn_qkv.bias, torch.float16 --> F32, shape = {3072} | |
| INFO:hf-to-gguf:blk.29.attn_qkv.weight, torch.float16 --> BF16, shape = {1024, 3072} | |
| INFO:hf-to-gguf:blk.29.attn_output.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:blk.29.attn_output.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:cls.norm.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:cls.norm.weight, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:cls.bias, torch.float16 --> F32, shape = {1024} | |
| INFO:hf-to-gguf:cls.weight, torch.float16 --> BF16, shape = {1024, 1024} | |
| INFO:hf-to-gguf:cls.output.bias, torch.float16 --> F32, shape = {1} | |
| INFO:hf-to-gguf:cls.output.weight, torch.float16 --> BF16, shape = {1024, 1} | |
| INFO:hf-to-gguf:token_types.weight, torch.float16 --> F32, shape = {1024, 3} | |
| INFO:hf-to-gguf:Set meta model | |
| INFO:hf-to-gguf:Set model parameters | |
| INFO:hf-to-gguf:gguf: context length = 8192 | |
| INFO:hf-to-gguf:gguf: embedding length = 1024 | |
| INFO:hf-to-gguf:gguf: feed forward length = [2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 2624, 4096, 4096] | |
| INFO:hf-to-gguf:gguf: head count = 16 | |
| WARNING:hf-to-gguf:Unknown RoPE type: default | |
| INFO:hf-to-gguf:gguf: rope scaling type = NONE | |
| INFO:hf-to-gguf:gguf: rope theta = 160000.0 | |
| INFO:hf-to-gguf:gguf: rope theta swa = 10000.0 | |
| INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-05 | |
| INFO:hf-to-gguf:gguf: layer norm epsilon = 1e-05 | |
| INFO:hf-to-gguf:gguf: file type = 32 | |
| INFO:hf-to-gguf:Set model quantization version | |
| INFO:hf-to-gguf:Set model tokenizer | |
| [transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. | |
| INFO:gguf.vocab:Adding 50009 merge(s). | |
| INFO:gguf.vocab:Setting special token type bos to 50281 | |
| INFO:gguf.vocab:Setting special token type eos to 50282 | |
| INFO:gguf.vocab:Setting special token type unk to 50280 | |
| INFO:gguf.vocab:Setting special token type sep to 50282 | |
| INFO:gguf.vocab:Setting special token type pad to 50283 | |
| WARNING:gguf.vocab:No handler for special token type cls with id 50281 - skipping | |
| INFO:gguf.vocab:Setting special token type mask to 50284 | |
| INFO:gguf.vocab:Setting add_bos_token to True | |
| WARNING:gguf.gguf_writer:Duplicated key name 'tokenizer.ggml.add_bos_token', overwriting it with new value True of type BOOL | |
| INFO:gguf.vocab:Setting add_eos_token to True | |
| WARNING:gguf.gguf_writer:Duplicated key name 'tokenizer.ggml.add_eos_token', overwriting it with new value True of type BOOL | |
| INFO:gguf.vocab:Setting add_sep_token to False | |
| WARNING:gguf.gguf_writer:Duplicated key name 'tokenizer.ggml.add_sep_token', overwriting it with new value False of type BOOL | |
| INFO:gguf.vocab:Setting normalizer_nfc to True | |
| INFO:gguf.vocab:Setting clean_spaces to True | |
| INFO:gguf.gguf_writer:Writing the following files: | |
| INFO:gguf.gguf_writer:upload-Laya/Laya-BF16.gguf: n_tensors = 201, total_size = 842.2M | |
| Writing: 0%| | 0.00/842M [00:00<?, ?byte/s][A | |
| Writing: 38%|ββββ | 318M/842M [00:01<00:01, 317Mbyte/s][A | |
| Writing: 77%|ββββββββ | 651M/842M [00:02<00:00, 325Mbyte/s][A Writing: 100%|ββββββββββ| 842M/842M [00:02<00:00, 326Mbyte/s] | |
| INFO:hf-to-gguf:Model successfully exported to upload-Laya/Laya-BF16.gguf | |
| + llama.cpp/build/bin/llama-quantize ./upload-Laya/Laya-BF16.gguf ./upload-Laya/Laya-Q8_0.gguf Q8_0 | |
| version: 0.9.0-dev (build 1, commit 0d69255) | |
| built with GNU 14.2.0 for Linux x86_64 | |
| llama_quantize: quantizing './upload-Laya/Laya-BF16.gguf' to './upload-Laya/Laya-Q8_0.gguf' as Q8_0 | |
| llama_model_loader: loaded meta data with 56 key-value pairs and 201 tensors from ./upload-Laya/Laya-BF16.gguf (version GGUF V3 (latest)) | |
| llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. | |
| llama_model_loader: - kv 0: general.architecture str = modern-bert | |
| llama_model_loader: - kv 1: general.type str = model | |
| llama_model_loader: - kv 2: general.name str = Laya | |
| llama_model_loader: - kv 3: general.size_label str = 421M | |
| llama_model_loader: - kv 4: general.license str = apache-2.0 | |
| llama_model_loader: - kv 5: general.tags arr[str,12] = ["laya", "system-one", "calibrated-de... | |
| llama_model_loader: - kv 6: modern-bert.block_count u32 = 30 | |
| llama_model_loader: - kv 7: modern-bert.context_length u32 = 8192 | |
| llama_model_loader: - kv 8: modern-bert.embedding_length u32 = 1024 | |
| llama_model_loader: - kv 9: modern-bert.feed_forward_length arr[i32,30] = [2624, 2624, 2624, 2624, 2624, 2624, ... | |
| llama_model_loader: - kv 10: modern-bert.attention.head_count u32 = 16 | |
| llama_model_loader: - kv 11: modern-bert.rope.freq_base f32 = 160000.000000 | |
| llama_model_loader: - kv 12: modern-bert.rope.freq_base_swa f32 = 10000.000000 | |
| llama_model_loader: - kv 13: modern-bert.attention.layer_norm_rms_epsilon f32 = 0.000010 | |
| llama_model_loader: - kv 14: modern-bert.attention.layer_norm_epsilon f32 = 0.000010 | |
| llama_model_loader: - kv 15: general.file_type u32 = 32 | |
| llama_model_loader: - kv 16: modern-bert.attention.causal bool = false | |
| llama_model_loader: - kv 17: modern-bert.classifier.pooling_type u32 = 1 | |
| llama_model_loader: - kv 18: modern-bert.classifier.activation str = gelu | |
| llama_model_loader: - kv 19: modern-bert.attention.sliding_window u32 = 128 | |
| llama_model_loader: - kv 20: modern-bert.attention.sliding_window_pattern u32 = 3 | |
| llama_model_loader: - kv 21: modern-bert.rope.scaling.type str = none | |
| llama_model_loader: - kv 22: modern-bert.vocab_size u32 = 50368 | |
| llama_model_loader: - kv 23: modern-bert.hidden_activation str = gelu | |
| llama_model_loader: - kv 24: modern-bert.decision.type str = laya | |
| llama_model_loader: - kv 25: modern-bert.decision.block_count u32 = 2 | |
| llama_model_loader: - kv 26: modern-bert.decision.max_head_tokens u32 = 192 | |
| llama_model_loader: - kv 27: modern-bert.decision.temperature.choice f32 = 1.636903 | |
| llama_model_loader: - kv 28: modern-bert.decision.temperature.score f32 = 1.251430 | |
| llama_model_loader: - kv 29: modern-bert.decision.temperature.noul f32 = 1.983400 | |
| llama_model_loader: - kv 30: modern-bert.decision.temperature.choice.3_5 f32 = 1.760152 | |
| llama_model_loader: - kv 31: modern-bert.decision.temperature.choice.6_10 f32 = 1.000016 | |
| llama_model_loader: - kv 32: modern-bert.decision.temperature.score.3_5 f32 = 1.251430 | |
| llama_model_loader: - kv 33: modern-bert.decision.temperature.noul.2 f32 = 1.983400 | |
| llama_model_loader: - kv 34: modern-bert.decision.temperature.choice.11 f32 = 0.100583 | |
| llama_model_loader: - kv 35: modern-bert.decision.temperature.choice.2 f32 = 1.906356 | |
| llama_model_loader: - kv 36: general.quantization_version u32 = 2 | |
| llama_model_loader: - kv 37: tokenizer.ggml.add_bos_token bool = true | |
| llama_model_loader: - kv 38: tokenizer.ggml.add_eos_token bool = true | |
| llama_model_loader: - kv 39: tokenizer.ggml.add_sep_token bool = false | |
| llama_model_loader: - kv 40: tokenizer.ggml.model str = gpt2 | |
| llama_model_loader: - kv 41: tokenizer.ggml.pre str = modern-bert | |
| llama_model_loader: - kv 42: tokenizer.ggml.tokens arr[str,50368] = ["|||IP_ADDRESS|||", "<|padding|>", "... | |
| llama_model_loader: - kv 43: tokenizer.ggml.token_type arr[i32,50368] = [4, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... | |
| llama_model_loader: - kv 44: tokenizer.ggml.merges arr[str,50009] = ["Δ Δ ", "Δ t", "Δ a", "h e", "i n... | |
| llama_model_loader: - kv 45: tokenizer.ggml.bos_token_id u32 = 50281 | |
| llama_model_loader: - kv 46: tokenizer.ggml.eos_token_id u32 = 50282 | |
| llama_model_loader: - kv 47: tokenizer.ggml.unknown_token_id u32 = 50280 | |
| llama_model_loader: - kv 48: tokenizer.ggml.seperator_token_id u32 = 50282 | |
| llama_model_loader: - kv 49: tokenizer.ggml.padding_token_id u32 = 50283 | |
| llama_model_loader: - kv 50: tokenizer.ggml.mask_token_id u32 = 50284 | |
| llama_model_loader: - kv 51: tokenizer.ggml.normalizer.nfc bool = true | |
| llama_model_loader: - kv 52: tokenizer.ggml.clean_spaces bool = true | |
| llama_model_loader: - kv 53: tokenizer.ggml.token_type_count u32 = 3 | |
| llama_model_loader: - kv 54: tokenizer.chat_template.systemone str = [CLS]{{ type }} question: {{ instruct... | |
| llama_model_loader: - kv 55: tokenizer.chat_templates arr[str,1] = ["systemone"] | |
| llama_model_loader: - type f32: 78 tensors | |
| llama_model_loader: - type bf16: 123 tensors | |
| [ 1/ 201] cls.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 2/ 201] cls.norm.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 3/ 201] cls.norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 4/ 201] cls.output.bias - [ 1, 1, 1, 1], type = f32, size = 0.000 MiB | |
| [ 5/ 201] cls.output.weight - [ 1024, 1, 1, 1], type = bf16, size = 0.002 MiB | |
| [ 6/ 201] cls.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 7/ 201] output_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 8/ 201] token_embd.weight - [ 1024, 50368, 1, 1], type = bf16, converting to q8_0 .. size = 98.38 MiB -> 52.26 MiB | |
| [ 9/ 201] token_embd_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 10/ 201] token_types.weight - [ 1024, 3, 1, 1], type = f32, size = 0.012 MiB | |
| [ 11/ 201] blk.0.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 12/ 201] blk.0.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 13/ 201] blk.0.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 14/ 201] blk.0.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 15/ 201] blk.0.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 16/ 201] blk.1.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 17/ 201] blk.1.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 18/ 201] blk.1.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 19/ 201] blk.1.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 20/ 201] blk.1.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 21/ 201] blk.1.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 22/ 201] blk.2.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 23/ 201] blk.2.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 24/ 201] blk.2.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 25/ 201] blk.2.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 26/ 201] blk.2.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 27/ 201] blk.2.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 28/ 201] blk.3.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 29/ 201] blk.3.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 30/ 201] blk.3.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 31/ 201] blk.3.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 32/ 201] blk.3.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 33/ 201] blk.3.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 34/ 201] blk.4.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 35/ 201] blk.4.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 36/ 201] blk.4.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 37/ 201] blk.4.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 38/ 201] blk.4.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 39/ 201] blk.4.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 40/ 201] blk.5.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 41/ 201] blk.5.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 42/ 201] blk.5.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 43/ 201] blk.5.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 44/ 201] blk.5.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 45/ 201] blk.5.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 46/ 201] blk.6.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 47/ 201] blk.6.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 48/ 201] blk.6.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 49/ 201] blk.6.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 50/ 201] blk.6.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 51/ 201] blk.6.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 52/ 201] blk.7.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 53/ 201] blk.7.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 54/ 201] blk.7.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 55/ 201] blk.7.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 56/ 201] blk.7.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 57/ 201] blk.7.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 58/ 201] blk.8.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 59/ 201] blk.8.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 60/ 201] blk.8.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 61/ 201] blk.8.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 62/ 201] blk.8.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 63/ 201] blk.8.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 64/ 201] blk.9.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 65/ 201] blk.9.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 66/ 201] blk.9.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 67/ 201] blk.9.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 68/ 201] blk.9.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 69/ 201] blk.9.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 70/ 201] blk.10.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 71/ 201] blk.10.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 72/ 201] blk.10.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 73/ 201] blk.10.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 74/ 201] blk.10.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 75/ 201] blk.10.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 76/ 201] blk.11.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 77/ 201] blk.11.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 78/ 201] blk.11.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 79/ 201] blk.11.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 80/ 201] blk.11.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 81/ 201] blk.11.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 82/ 201] blk.12.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 83/ 201] blk.12.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 84/ 201] blk.12.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 85/ 201] blk.12.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 86/ 201] blk.12.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 87/ 201] blk.12.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 88/ 201] blk.13.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 89/ 201] blk.13.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 90/ 201] blk.13.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 91/ 201] blk.13.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 92/ 201] blk.13.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 93/ 201] blk.13.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 94/ 201] blk.14.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 95/ 201] blk.14.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 96/ 201] blk.14.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 97/ 201] blk.14.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 98/ 201] blk.14.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 99/ 201] blk.14.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 100/ 201] blk.15.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 101/ 201] blk.15.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 102/ 201] blk.15.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 103/ 201] blk.15.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 104/ 201] blk.15.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 105/ 201] blk.15.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 106/ 201] blk.16.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 107/ 201] blk.16.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 108/ 201] blk.16.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 109/ 201] blk.16.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 110/ 201] blk.16.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 111/ 201] blk.16.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 112/ 201] blk.17.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 113/ 201] blk.17.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 114/ 201] blk.17.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 115/ 201] blk.17.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 116/ 201] blk.17.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 117/ 201] blk.17.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 118/ 201] blk.18.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 119/ 201] blk.18.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 120/ 201] blk.18.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 121/ 201] blk.18.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 122/ 201] blk.18.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 123/ 201] blk.18.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 124/ 201] blk.19.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 125/ 201] blk.19.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 126/ 201] blk.19.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 127/ 201] blk.19.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 128/ 201] blk.19.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 129/ 201] blk.19.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 130/ 201] blk.20.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 131/ 201] blk.20.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 132/ 201] blk.20.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 133/ 201] blk.20.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 134/ 201] blk.20.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 135/ 201] blk.20.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 136/ 201] blk.21.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 137/ 201] blk.21.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 138/ 201] blk.21.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 139/ 201] blk.21.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 140/ 201] blk.21.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 141/ 201] blk.21.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 142/ 201] blk.22.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 143/ 201] blk.22.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 144/ 201] blk.22.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 145/ 201] blk.22.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 146/ 201] blk.22.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 147/ 201] blk.22.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 148/ 201] blk.23.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 149/ 201] blk.23.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 150/ 201] blk.23.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 151/ 201] blk.23.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 152/ 201] blk.23.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 153/ 201] blk.23.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 154/ 201] blk.24.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 155/ 201] blk.24.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 156/ 201] blk.24.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 157/ 201] blk.24.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 158/ 201] blk.24.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 159/ 201] blk.24.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 160/ 201] blk.25.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 161/ 201] blk.25.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 162/ 201] blk.25.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 163/ 201] blk.25.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 164/ 201] blk.25.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 165/ 201] blk.25.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 166/ 201] blk.26.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 167/ 201] blk.26.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 168/ 201] blk.26.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 169/ 201] blk.26.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 170/ 201] blk.26.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 171/ 201] blk.26.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 172/ 201] blk.27.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 173/ 201] blk.27.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 174/ 201] blk.27.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 175/ 201] blk.27.ffn_down.weight - [ 2624, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 5.12 MiB -> 2.72 MiB | |
| [ 176/ 201] blk.27.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 177/ 201] blk.27.ffn_up.weight - [ 1024, 5248, 1, 1], type = bf16, converting to q8_0 .. size = 10.25 MiB -> 5.45 MiB | |
| [ 178/ 201] blk.28.attn_norm.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 179/ 201] blk.28.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 180/ 201] blk.28.attn_output.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 181/ 201] blk.28.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 182/ 201] blk.28.attn_qkv.bias - [ 3072, 1, 1, 1], type = f32, size = 0.012 MiB | |
| [ 183/ 201] blk.28.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 184/ 201] blk.28.ffn_down.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 185/ 201] blk.28.ffn_down.weight - [ 4096, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB | |
| [ 186/ 201] blk.28.ffn_norm.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 187/ 201] blk.28.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 188/ 201] blk.28.ffn_up.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MiB | |
| [ 189/ 201] blk.28.ffn_up.weight - [ 1024, 4096, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB | |
| [ 190/ 201] blk.29.attn_norm.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 191/ 201] blk.29.attn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 192/ 201] blk.29.attn_output.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 193/ 201] blk.29.attn_output.weight - [ 1024, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 2.00 MiB -> 1.06 MiB | |
| [ 194/ 201] blk.29.attn_qkv.bias - [ 3072, 1, 1, 1], type = f32, size = 0.012 MiB | |
| [ 195/ 201] blk.29.attn_qkv.weight - [ 1024, 3072, 1, 1], type = bf16, converting to q8_0 .. size = 6.00 MiB -> 3.19 MiB | |
| [ 196/ 201] blk.29.ffn_down.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 197/ 201] blk.29.ffn_down.weight - [ 4096, 1024, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB | |
| [ 198/ 201] blk.29.ffn_norm.bias - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 199/ 201] blk.29.ffn_norm.weight - [ 1024, 1, 1, 1], type = f32, size = 0.004 MiB | |
| [ 200/ 201] blk.29.ffn_up.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MiB | |
| [ 201/ 201] blk.29.ffn_up.weight - [ 1024, 4096, 1, 1], type = bf16, converting to q8_0 .. size = 8.00 MiB -> 4.25 MiB | |
| llama_model_quantize_impl: model size = 803.22 MiB (16.00 BPW) | |
| llama_model_quantize_impl: quant size = 426.88 MiB (8.51 BPW) | |
| llama_quantize: quantize time = 813.54 ms | |
| llama_quantize: total time = 813.54 ms | |
| + echo Laya-BF16.gguf | |
| + echo Laya-Q8_0.gguf | |