Instructions to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
Use Docker
docker model run hf.co/SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
- Ollama
How to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with Ollama:
ollama run hf.co/SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with Docker Model Runner:
docker model run hf.co/SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
- Lemonade
How to use SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Meta-Llama-3-8B-Instruct-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Model Details (GGUF 4bit)
Meta developed and released the Meta Llama 3 family of large language models (LLMs), a collection of pretrained and instruction tuned generative text models in 8 and 70B sizes. The Llama 3 instruction tuned models are optimized for dialogue use cases and outperform many of the available open source chat models on common industry benchmarks. Further, in developing these models, we took great care to optimize helpfulness and safety.
Model developers Meta
Variations Llama 3 comes in two sizes — 8B and 70B parameters — in pre-trained and instruction tuned variants.
Input Models input text only.
Output Models generate text and code only.
Model Architecture Llama 3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
| Training Data | Params | Context length | GQA | Token count | Knowledge cutoff | |
| Llama 3 | A new mix of publicly available online data. | 8B | 8k | Yes | 15T+ | March, 2023 |
| 70B | 8k | Yes | December, 2023 |
Llama 3 family of models. Token counts refer to pretraining data only. Both the 8 and 70B versions use Grouped-Query Attention (GQA) for improved inference scalability.
Model Release Date April 18, 2024.
Status This is a static model trained on an offline dataset. Future versions of the tuned models will be released as we improve model safety with community feedback.
License A custom commercial license is available at: https://llama.meta.com/llama3/license
Where to send questions or comments about the model Instructions on how to provide feedback or comments on the model can be found in the model README. For more technical information about generation parameters and recipes for how to use Llama 3 in applications, please go here.
Intended Use
Intended Use Cases Llama 3 is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
Out-of-scope Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3 Community License. Use in languages other than English**.
**Note: Developers may fine-tune Llama 3 models for languages beyond English provided they comply with the Llama 3 Community License and the Acceptable Use Policy.
First install the package
Run one of the following commands, according to your system:
# Base ctransformers with no GPU acceleration
pip install llama-cpp-python
# With NVidia CUDA acceleration
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
# Or with OpenBLAS acceleration
CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python
# Or with CLBLast acceleration
CMAKE_ARGS="-DLLAMA_CLBLAST=on" pip install llama-cpp-python
# Or with AMD ROCm GPU acceleration (Linux only)
CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python
# Or with Metal GPU acceleration for macOS systems only
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python
# In windows, to set the variables CMAKE_ARGS in PowerShell, follow this format; eg for NVidia CUDA:
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on"
pip install llama-cpp-python
Download the model into current directory
from huggingface_hub import hf_hub_download
REPO_ID = "SalmanFaroz/Meta-Llama-3-8B-Instruct-GGUF"
FILENAME = "Q4_K_M.gguf"
hf_hub_download(repo_id=REPO_ID, filename=FILENAME,local_dir="./")
Simple llama-cpp-python example code
from llama_cpp import Llama
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
model_path="./Q4_K_M.gguf", # Download the model file first
n_ctx=4096, # The max sequence length to use - note that longer sequence lengths require much more resources
n_threads=8, # The number of CPU threads to use, tailor to your system and the resulting performance
n_gpu_layers=35 # The number of layers to offload to GPU, if you have GPU acceleration available
)
prompt = "Tell me about AI"
output = llm(
f'''[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
{prompt}[/INST]
''', # Prompt
max_tokens=100, # Generate up to 512 tokens
stop=["</s>"], # Example stop token - not necessarily correct for this specific model! Please check before using.
echo=True , # Whether to echo the prompt
temperature=0.001
)
- Downloads last month
- 52
4-bit