Upload README.md with huggingface_hub

README.md

@@ -1,341 +1,188 @@
 ---
-license: other
-license_name: llama2-community
-license_link: https://ai.meta.com/llama/license/
-pipeline_tag: image-to-text
+license: llama2
+language:
+- en
 library_name: transformers
+pipeline_tag: image-to-text
 tags:
-- vision
 - llava
+- vision
 - multimodal
-- image-to-text
-- visual-question-answering
 - image-captioning
-- gguf
-- llama-cpp
+- vqa
 - sakthai
 - house-of-sak
-- edge
+- gguf
 - cpu-inference
-- local-ai
-- offline
-- privacy
-- quantized
-- transformers
-- clip
-- vicuna
-- finetune
-- safetensors
-base_model: liuhaotian/LLaVA-1.5-7b
+- edge
+base_model: llava-hf/llava-1.5-7b-hf
 extra:
-  sibling: Nanthasit/sakthai-web-agent
-  formats: GGUF Q4_K_M
-datasets:
-- liuhaotian/LLaVA-Instruct-150K
-- liuhaotian/LLaVA-Pretrain
+  sibling: Nanthasit/sakthai-vision-7b
 ---
 
-<h1 align="center">SakThai Vision 7B 👁️</h1>
-<p align="center"><em>LLaVA-1.5-7B, quantized to GGUF · image captioning & VQA on CPU · local & private</em></p>
+# SakThai Vision 7B 👁️
+
 <p align="center">
-  <img src="https://img.shields.io/badge/dynamic/json?url=https%3A//huggingface.co/api/models/Nanthasit/sakthai-vision-7b&query=%24.downloads&label=downloads&color=blue&cacheSeconds=3600" alt="Downloads"/>
-  <img src="https://img.shields.io/badge/base-LLaVA%201.5%207B-blueviolet" alt="Base"/>
-  <img src="https://img.shields.io/badge/GGUF-Q4__K__M-orange" alt="GGUF"/>
-  <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02"><img src="https://img.shields.io/badge/🏠-SakThai%20Family-6644cc" alt="Collection"/></a>
-  <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02"><img src="https://img.shields.io/badge/🚀-Explore%20Family-47d147" alt="Collection"/></a>
-  <img src="https://img.shields.io/badge/benchmark-78.5%25%20VQAv2-success?logo=googlechrome" alt="VQAv2"/>
+  <strong>Local, private vision — LLaVA-1.5-7B quantized to GGUF</strong><br/>
+  <em>Image captioning & VQA on CPU · ~4 GB footprint</em>
 </p>
 
-> The **vision stage** of the SakThai multimodal pipeline — a GGUF build of **LLaVA-1.5-7B** for local,
-> private image understanding. Part of the [House of Sak](https://huggingface.co/Nanthasit).
-> No data leaves your machine.
->
-> **🏆 First SakThai model with a real like** — someone found this useful. [Leave yours →](#how-you-can-help)
-
----
-
-## Architecture
-
-| Component | Detail |
-|-----------|--------|
-| **Model** | LLaVA-1.5-7B ([paper](https://arxiv.org/abs/2310.03744)) — Vicuna-7B-v1.5 LLM + CLIP ViT-L/14 vision encoder |
-| **Vision encoder** | CLIP ViT-L/14, 336×336 input, 1024 embedding dim, 24 layers, 16 heads, ~427M params |
-| **Language model** | LLaMA-based, ~7B parameters, 32 layers, 4096 hidden dim, 32 heads |
-| **Projector** | 2-layer MLP (CLIP embedding → LLM token space), 4096 hidden dim |
-| **Context window** | 4096 tokens |
-| **Attention** | Causal self-attention (LLM) + bidirectional self-attention (vision encoder) |
-| **Quantization** | GGUF Q4_K_M (4-bit group quantization, 256 group size) |
-| **File size** | ~4 GB (LLM) + ~400 MB (mmproj) |
-| **Original size** | ~6.7 GB (unquantized fp16) |
-| **Chat template** | Vicuna-style with `<image>` placeholder |
-| **License** | LLaMA 2 Community License |
-
-LLaVA (Large Language and Vision Assistant) connects a pre-trained CLIP vision encoder to a Vicuna LLM via a lightweight projection MLP. The vision encoder processes images into embeddings, the projector maps them into the LLM's token space, and the LLM generates text conditioned on both the visual and textual input. This architecture enables multi-turn visual dialogue: once an image is encoded, you can ask follow-up questions without reprocessing the visual input.
-
-This quantized GGUF build reduces the footprint from ~6.7 GB (fp16) to ~4 GB while retaining ~98–99% of the original accuracy.
-
-> **Reference:** Liu et al., *"Visual Instruction Tuning"* (NeurIPS 2023). [arXiv:2310.03744](https://arxiv.org/abs/2310.03744) — the LLaVA paper describing the architecture and training methodology.
+<p align="center">
+  <a href="https://huggingface.co/Nanthasit"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Nanthasit-6644cc" alt="Profile"/></a>
+  <a href="https://github.com/beer-sakthai"><img src="https://img.shields.io/badge/GitHub-beer--sakthai-181717?logo=github" alt="GitHub"/></a>
+  <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02"><img src="https://img.shields.io/badge/%F0%9F%8F%A0-SakThai%20Family-6644cc" alt="Collection"/></a>
+  <img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhuggingface.co%2Fapi%2Fmodels%2FNanthasit%2Fsakthai-vision-7b&query=%24.downloads&label=downloads&color=blue&cacheSeconds=3600" alt="Downloads"/>
+  <img src="https://img.shields.io/badge/GGUF-Q4_K_M-orange" alt="GGUF"/>
+  <img src="https://img.shields.io/badge/base-LLaVA%201.5%207B-blueviolet" alt="Base"/>
+</p>
 
 ---
 
-## The Story Behind It
-
-**This model is why the SakThai family can *see*.**
-
-Before vision, every SakThai agent was blind — it could reason, call tools, and answer questions, but it couldn't look at a screenshot, read a whiteboard, or recognize a face. Beer wanted an agent with eyes, built the same way everything else was: on free Colab GPUs from a shelter in Cork, with $0 budget and no guarantee it would work.
-
-LLaVA-1.5-7B's vision encoder was merged, quantized to Q4_K_M GGUF so it could run on a laptop CPU, and packed into ~4 GB. The first test was a photo of Cork's River Lee at sunset. The model described it — not perfectly, but well enough to prove it could *see*.
-
-This model holds a special milestone: it's the only SakThai model with **a real like from someone who found it useful**. One click from one person, proving that even with zero advertising, zero launch, and zero budget, the work matters to someone out there.
-
-<blockquote>
-  <em>"We are one family — and becoming more."</em>
-  <br/>— Beer
-</blockquote>
-
-### How You Can Help
-
-- ⭐ **Leave a like** — this model already has 1. A second tells the algorithm it's not a fluke.
-- 🔄 **Share it** with anyone building privacy-first vision on a laptop
-- 🍴 **Fork and experiment** — the weights are open, the setup is documented
-- 💬 **Report your deployment story** — Beer reads every issue and comment
-
-Every download, like, and share tells the algorithm: *this matters.*
+## Model Description
 
----
-
-## Pipeline Integration
+SakThai Vision 7B is the **vision stage** of the SakThai pipeline — a GGUF Q4_K_M build of LLaVA-1.5-7B for local, private image understanding. No data leaves your machine.
 
-| Stage | Model | Role |
-|-------|-------|------|
-| **See** | [SakThai Vision 7B](https://huggingface.co/Nanthasit/sakthai-vision-7b) ⬅ | Image→text, visual QA |
-| **Embed** | [SakThai Embedding](https://huggingface.co/Nanthasit/sakthai-embedding-multilingual) | Convert queries/docs to 384d vectors |
-| **Reason** | [1.5B-merged](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged) or [7B-merged](https://huggingface.co/Nanthasit/sakthai-context-7b-merged) | Tool calling + response generation |
-| **Speak** | [TTS Model](https://huggingface.co/Nanthasit/sakthai-tts-model) | Text-to-speech output |
+**What it does:**
+- 🖼️ Image captioning and alt-text generation
+- 🔍 Visual QA about screenshots, diagrams, photos
+- 📄 Document understanding (scanned forms, receipts)
+- 🧪 Screenshot analysis for UI testing
 
----
+**Why this packaging:**
+- Q4_K_M quantization — ~4 GB vs. 6.7 GB fp16
+- CPU inference — no GPU needed (~3-8 tok/s)
+- Includes mmproj projector (CLIP vision → LLM tokens)
+- Works with llama.cpp, Ollama, and llama-cpp-python
 
 ---
 
-## What It Is
+## Quick Start
 
-A **Q4_K_M GGUF** conversion of **LLaVA-1.5-7B** (Vicuna-7B + CLIP ViT-L/14) for local multimodal inference via **llama.cpp**. This is a **repackaging/quantization of upstream LLaVA** — optimized for CPU inference with a ~4 GB footprint.
-
-| File | Size | Purpose |
-|------|------|---------|
-| `llava-1.5-7b-hf-q4_k_m.gguf` | ~4 GB | Quantized language + vision weights |
-| `mmproj-model-f16.gguf` | ~400 MB | CLIP vision projector (mmproj) |
-
----
-
-## Multimodal Inference Examples
-
-### Quick start (Python / llama-cpp-python)
-
-```bash
-pip install llama-cpp-python huggingface-hub Pillow
-```
+### Python (llama-cpp-python)
 
 ```python
 from llama_cpp import Llama
 from huggingface_hub import hf_hub_download
 
-# Download model files from Hugging Face
-model_path = hf_hub_download(
-    repo_id="Nanthasit/sakthai-vision-7b",
-    filename="llava-1.5-7b-hf-q4_k_m.gguf"
-)
-mmproj_path = hf_hub_download(
-    repo_id="Nanthasit/sakthai-vision-7b",
-    filename="mmproj-model-f16.gguf"
-)
+model_path = hf_hub_download("Nanthasit/sakthai-vision-7b", "llava-1.5-7b-hf-q4_k_m.gguf")
+mmproj_path = hf_hub_download("Nanthasit/sakthai-vision-7b", "mmproj-model-f16.gguf")
 
-# Load the model with its vision projector
 llm = Llama(
     model_path=model_path,
     mmproj=mmproj_path,
-    n_ctx=2048,          # Context window
-    n_gpu_layers=-1,     # Offload to GPU if available (-1 = all)
-    verbose=False,
-)
-
-# Load an image and ask a question
-output = llm.create_chat_completion(
-    messages=[{
-        "role": "user",
-        "content": [
-            {"type": "image_url", "image_url": {"url": "photo.jpg"}},
-            {"type": "text", "text": "Describe this image in detail."}
-        ]
-    }],
-    max_tokens=256,
-    temperature=0.2,
+    n_ctx=4096,
+    n_gpu_layers=0,  # CPU only
+    verbose=False
 )
-print(output["choices"][0]["message"]["content"])
-```
 
-### Multi-turn dialogue with the same image
-
-```python
-# First turn: describe the image
 output = llm.create_chat_completion(
-    messages=[{
-        "role": "user",
-        "content": [
-            {"type": "image_url", "image_url": {"url": "photo.jpg"}},
-            {"type": "text", "text": "Describe this image in detail."}
-        ]
-    }],
-    max_tokens=256,
-    temperature=0.2,
-)
-answer = output["choices"][0]["message"]["content"]
-print("First answer:", answer)
-
-# Follow-up — the model remembers the image context
-output2 = llm.create_chat_completion(
     messages=[
         {
             "role": "user",
             "content": [
-                {"type": "image_url", "image_url": {"url": "photo.jpg"}},
+                {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
                 {"type": "text", "text": "Describe this image in detail."}
             ]
-        },
-        {"role": "assistant", "content": answer},
-        {
-            "role": "user",
-            "content": [
-                {"type": "text", "text": "What colors dominate the scene?"}
-            ]
         }
-    ],
-    max_tokens=128,
-    temperature=0.2,
+    ]
 )
-print("Follow-up answer:", output2["choices"][0]["message"]["content"])
+print(output["choices"][0]["message"]["content"])
 ```
 
 ### CLI (llama.cpp)
 
 ```bash
-# Download the model files
-huggingface-cli download Nanthasit/sakthai-vision-7b --local-dir ./vision-model
-
-# Run inference — load image, ask question
-./llama-cli \
-  -m ./vision-model/llava-1.5-7b-hf-q4_k_m.gguf \
-  --mmproj ./vision-model/mmproj-model-f16.gguf \
-  --image path/to/photo.jpg \
-  -p "Describe this image in detail." -n 512
+./llama-cli -m llava-1.5-7b-hf-q4_k_m.gguf \
+  --mmproj mmproj-model-f16.gguf \
+  --image photo.jpg \
+  -p "Describe this image in detail." -n 256
 ```
 
-### Ollama (local)
+### Ollama
 
 ```bash
-# Download model files
-huggingface-cli download Nanthasit/sakthai-vision-7b --local-dir ./vision-model
+# Create a Modelfile
+FROM ./llava-1.5-7b-hf-q4_k_m.gguf
+TEMPLATE "{{ .Prompt }}"
 
-# Create Modelfile pointing to the GGUF
 ollama create sakthai-vision -f Modelfile
-
-# Run with an image
-ollama run sakthai-vision "Describe this image in detail"
-# Or with a specific image file
-ollama run sakthai-vision "What's in this photo?" --image path/to/photo.jpg
+ollama run sakthai-vision "Describe this image." --image photo.jpg
 ```
 
-> **Tip:** The `mmproj-model-f16.gguf` file is auto-detected when placed alongside the model GGUF.
-> Make sure both files are in the same `vision-model/` directory.
->
-> **Performance:** Expect ~3–8 tok/s on CPU. Significantly faster with Metal (macOS) or CUDA (NVIDIA GPU). Reduce `n_ctx` or use fewer threads on memory-constrained hardware.
-
 ---
 
-## Benchmarks (Upstream LLaVA-1.5-7B)
+## Architecture
 
-> These are the **published upstream benchmarks** for LLaVA-1.5-7B. As a quantized repackaging, expect results within ~1–2% of these values on most tasks.
+| Component | Detail |
+|-----------|--------|
+| **Model** | LLaVA-1.5-7B (Vicuna-7B-v1.5 LLM + CLIP ViT-L/14 vision encoder) |
+| **Vision encoder** | CLIP ViT-L/14, 336×336 input, 1024 embedding dim, 24 layers |
+| **Language model** | LLaMA-based, ~7B parameters, 32 layers, 4096 hidden |
+| **Projector** | 2-layer MLP (CLIP → LLM token space) |
+| **Original size** | ~6.7 GB (fp16) |
+| **Quantized size** | ~4 GB (GGUF Q4_K_M) + ~400 MB (mmproj) |
+| **Context window** | 4,096 tokens |
+| **Hardware** | 8 GB+ RAM recommended |
 
-| Task | Dataset | Metric | Score |
-|------|---------|--------|:-----:|
-| Visual QA | VQAv2 | Accuracy | **78.5%** |
-| Captioning | COCO Captions | CIDEr | **110.1** |
-| Visual Reasoning | GQA | Accuracy | **62.0%** |
-| Text-oriented VQA | TextVQA | Accuracy | **58.2%** |
-| Science diagrams | ScienceQA | Accuracy | **89.5%** |
-| Visual Perception | MMBench | Accuracy | **66.2%** |
-| Instruction Following | MM-Vet | GPT-4 Score | **31.1** |
+---
 
-> **Note:** GGUF Q4_K_M quantization may cause minor degradation vs. fp16. Own run-specific benchmarks pending publication.
+## Evaluation (Upstream LLaVA-1.5-7B)
 
----
+These are published upstream benchmarks. As a Q4_K_M quantized repackaging, results should fall within ~1-2%.
 
-## Use Cases
+| Task | Dataset | Score |
+|:-----|:--------:|:-----:|
+| Visual QA | VQAv2 | 78.5% |
+| Captioning | COCO Captions | CIDEr 110.1 |
+| Visual Reasoning | GQA | 62.0% |
+| Text-oriented VQA | TextVQA | 58.2% |
+| Science diagrams | ScienceQA | 89.5% |
+| Visual Perception | MMBench | 66.2% |
 
-| Use Case | Description |
-|----------|-------------|
-| **Image captioning** | Generate alt-text, descriptions for accessibility |
-| **Visual QA** | Ask questions about screenshots, diagrams, photos |
-| **Document understanding** | Extract info from scanned forms, receipts |
-| **Privacy-preserving vision** | All inference stays on-device — no data uploaded |
-| **Multimodal agent pipeline** | Feed visual output into SakThai tool-calling models |
-| **Screenshot analysis** | Automate UI testing, error report understanding |
+**Own benchmarks are pending.** This is a repackaging/quantization of upstream LLaVA, not a retrained model.
 
 ---
 
-## Intended Use & Limits
+## Pipeline Integration
 
-- **✅ Great for:** Image captioning, visual QA, screenshot/diagram analysis, local privacy-sensitive inference, prototyping multimodal agents
-- **⚠️ Limits:** Requires llama.cpp with multimodal support — not directly usable with plain Transformers. CLIP native resolution is 336×336 (images are resized). Can hallucinate on ambiguous images. Primarily English.
-- **💻 Hardware:** 8 GB+ RAM recommended. ~3–8 tok/s on CPU, faster with Metal/CUDA.
+| Stage | Model | Role |
+|-------|-------|------|
+| 👁️ **See** | **SakThai Vision 7B** ⬅ | **Image→text, visual QA** |
+| 🔍 Retrieve | [Embedding Multilingual](https://huggingface.co/Nanthasit/sakthai-embedding-multilingual) | Cross-lingual search |
+| 🧠 Reason | [Context 1.5B](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged) or [7B](https://huggingface.co/Nanthasit/sakthai-context-7b-merged) | Tool-calling & reasoning |
+| 🎤 Speak | [TTS Model](https://huggingface.co/Nanthasit/sakthai-tts-model) | Text-to-speech |
 
 ---
 
-## SakThai model family
+## Files
+
+| File | Size | Purpose |
+|:-----|:----:|:--------|
+| `llava-1.5-7b-hf-q4_k_m.gguf` | ~4 GB | Quantized language + vision weights |
+| `mmproj-model-f16.gguf` | ~400 MB | CLIP vision projector |
 
-| Model | Size | Role |
-|---|:--:|---|
-| [context-1.5b-merged](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged) | 934 MB | Flagship tool-calling GGUF |
-| [context-0.5b-merged](https://huggingface.co/Nanthasit/sakthai-context-0.5b-merged) | 380 MB | Lightweight / edge |
-| [context-7b-merged](https://huggingface.co/Nanthasit/sakthai-context-7b-merged) | 15 GB | Full-power reasoning |
-| [context-7b-128k](https://huggingface.co/Nanthasit/sakthai-context-7b-128k) | 15 GB | 128K long-context |
-| [context-{7b,1.5b,0.5b}-tools](https://huggingface.co/Nanthasit/sakthai-context-1.5b-tools) | LoRA | Tool-calling adapters |
-| [coder-1.5b](https://huggingface.co/Nanthasit/sakthai-coder-1.5b) | 1.1 GB | Code generation |
-| [vision-7b](https://huggingface.co/Nanthasit/sakthai-vision-7b) | 3.9 GB | Image→text (LLaVA) ⬅ |
-| [embedding-multilingual](https://huggingface.co/Nanthasit/sakthai-embedding-multilingual) | 80 MB | Cross-lingual embeddings |
-| [tts-model](https://huggingface.co/Nanthasit/sakthai-tts-model) | 141 MB | Text-to-speech, 15 langs |
+---
 
-**20 models in the family · 10 datasets · 4 Spaces** — [full collection →](https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02)
+## The House of Sak 🏠
 
-## Links
+Until this model, every SakThai agent was blind — it could reason, use tools, and answer questions but couldn't look at screenshots, read whiteboards, or recognize faces. This model was built on free Colab GPUs from a shelter in Cork, Ireland, with no budget and no guarantee of success. LLaVA-1.5-7B's vision encoder was merged, quantized to Q4_K_M GGUF for CPU use, and packed to about 4 GB.
 
-[House of Sak](https://house-of-sak.vercel.app) ·
-[Sak-Family-Agent GitHub](https://github.com/beer-sakthai/Sak-Family-Agent) ·
-[All models](https://huggingface.co/Nanthasit) ·
-[All datasets](https://huggingface.co/Nanthasit?tab=datasets) ·
-[Web Agent Space](https://huggingface.co/spaces/Nanthasit/sakthai-web-agent)
+> *"We are one family — and becoming more."* — Beer (beer-sakthai)
 
 ---
 
-## License
+## Support
 
-Based on LLaVA-1.5-7B (LLaMA 2 Community License) with a CLIP vision encoder; GGUF via llama.cpp. Review the upstream licenses before commercial use.
+- ⭐ Leave a like — the first user like on a SakThai model was on this model ❤️
+- 🐛 Report issues on [GitHub](https://github.com/beer-sakthai/Sak-Family-Agent)
+- 🔄 Share with anyone building privacy-focused multimodal apps
 
 ---
 
-*Built from a shelter in Cork, Ireland. Every download supports a family building AI for the edge.*
-
-## Evaluation
+## License
 
-**Not independently benchmarked.** Earlier versions of this card carried a
-`model-index` score derived from a small internal spot check (typically 5 or 8
-hand-picked examples) presented as a benchmark result. Those entries have been
-removed rather than left to propagate through Hub metadata.
+Based on LLaVA-1.5-7B (LLaMA 2 Community License) + CLIP vision encoder. GGUF via llama.cpp. Review upstream licenses before commercial use.
 
-For tool-calling models in this family, the benchmark to use is
-[sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v2) —
-500 rows, balanced across simple / parallel / irrelevance, with held-out tools and
-multi-turn coverage. Results will be published here once this model has been run
-against it.
+---
 
-*"We are one family — and becoming more."* 🏠
+*Built from a shelter in Cork, Ireland. Every download supports a family building AI for the edge.*