Saiga GGUF
Collection
Russian fine-tunes of different base LLMs in the GGUF format compatible with llama.cpp • 8 items • Updated • 39
How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IlyaGusev/saiga_nemo_12b_gguf:# Run inference directly in the terminal:
llama cli -hf IlyaGusev/saiga_nemo_12b_gguf:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf IlyaGusev/saiga_nemo_12b_gguf:# Run inference directly in the terminal:
./llama-cli -hf IlyaGusev/saiga_nemo_12b_gguf:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf IlyaGusev/saiga_nemo_12b_gguf:# Run inference directly in the terminal:
./build/bin/llama-cli -hf IlyaGusev/saiga_nemo_12b_gguf:Use Docker
docker model run hf.co/IlyaGusev/saiga_nemo_12b_gguf:Quick Links
Llama.cpp compatible versions of an original 12B model.
Download one of the versions, for example saiga_nemo_12b.Q4_K_M.gguf.
wget https://huggingface.co/IlyaGusev/saiga_nemo_12b_gguf/resolve/main/saiga_nemo_12b.Q4_K_M.gguf
Download interact_llama3_llamacpp.py
wget https://raw.githubusercontent.com/IlyaGusev/rulm/master/self_instruct/src/interact_llama3_llamacpp.py
How to run:
pip install llama-cpp-python fire
python3 interact_llama3_llamacpp.py saiga_nemo_12b.Q4_K_M.gguf
System requirements:
- 15GB RAM for q8_0 and less for smaller quantizations
- Downloads last month
- 4,396
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf IlyaGusev/saiga_nemo_12b_gguf:# Run inference directly in the terminal: llama cli -hf IlyaGusev/saiga_nemo_12b_gguf: