Nanthasit commited on
Commit
3ea38cd
·
verified ·
1 Parent(s): 8a91850

docs: improve SakThai Vision 7B model card — remove duplicate sections, add story, examples, architecture, quick start

Browse files
Files changed (1) hide show
  1. README.md +109 -17
README.md CHANGED
@@ -42,8 +42,6 @@ model-index:
42
  name: Verified Health
43
  ---
44
 
45
- # SakThai Vision 7B — Multimodal Tool-Use
46
-
47
  <p align="center">
48
  <strong>Image understanding + tool-use grounding for Qwen2-VL</strong><br/>
49
  <em>SakThai multimodal adapter · part of the <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02">SakThai Model Family</a></em>
@@ -60,24 +58,72 @@ model-index:
60
 
61
  ---
62
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  ## Model Description
64
 
65
- SakThai Vision 7B is a **multimodal understanding model** focused on images that need action, not just captioning. It is built for:
 
66
 
67
  - Visual question answering with image context
68
  - Screenshot parsing and structured extraction
69
  - Tool-use grounding from image content
70
  - Multimodal assistant-style reasoning
71
 
72
- **Important naming note:** This repo is branded as “Vision 7B” for family consistency, but the underlying architecture is **Qwen2-VL-2B-Instruct** with SakThai training; the name describes capability scope, not exact parameter count.
 
 
 
 
73
 
74
- ## Intended Use
 
75
 
76
- - Parse UI screenshots into structured actions or extracted fields
77
- - Answer questions about diagrams, documents, and photos
78
- - Ground tool calls in visual inputs when combined with a text tool layer
79
 
80
- ## How to Use
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
  ```python
83
  from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
@@ -90,36 +136,80 @@ messages = [
90
  {
91
  "role": "user",
92
  "content": [
93
- {"type": "image", "image": "your_image.jpg"},
94
- {"type": "text", "text": "<img> Describe what is shown and any actionable details."}
95
  ]
96
  }
97
  ]
98
 
99
  text = processor.apply_chat_template(messages, tokenize=False)
100
- inputs = processor(text=[text], images=["your_image.jpg"], return_tensors="pt")
101
  outputs = model.generate(**inputs, max_new_tokens=256)
102
  print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
103
  ```
104
 
105
  ### Save locally for reuse
 
106
  ```python
107
  model.save_pretrained("./sakthai-vision-7b")
108
  processor.save_pretrained("./sakthai-vision-7b")
109
  ```
110
 
111
- ## Training Details
 
 
 
 
 
 
 
112
 
113
- - **Base model:** Qwen/Qwen2-VL-2B-Instruct
114
- - **Training data:** Nanthasit/sakthai-combined-v7, Nanthasit/sakthai-bench-v2
115
- - **License:** Apache 2.0
116
- - **Primary use case:** image-text-to-text with tool-use style outputs
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
117
 
118
  ## Benchmarks
119
 
120
  | Dataset | Task | Note |
121
  |:--------|:-----|:-----|
122
  | SakThai Bench v2 | Multimodal Understanding | Card-level health indicator: 5 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
 
124
  ## Limitations
125
 
@@ -128,6 +218,8 @@ processor.save_pretrained("./sakthai-vision-7b")
128
  - Benchmarks are limited; treat reported metrics as indicative, not conclusive.
129
  - No standalone tool-execution layer included; combine with a separate tool router for agentic use.
130
 
 
 
131
  ## Citation
132
 
133
  ```bibtex
 
42
  name: Verified Health
43
  ---
44
 
 
 
45
  <p align="center">
46
  <strong>Image understanding + tool-use grounding for Qwen2-VL</strong><br/>
47
  <em>SakThai multimodal adapter · part of the <a href="https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02">SakThai Model Family</a></em>
 
58
 
59
  ---
60
 
61
+ > The **vision** branch of the SakThai family — a small multimodal model built for screenshots,
62
+ > documents, diagrams, and tool grounding instead of generic captioning.
63
+ > It is tuned to return *structured, actionable* descriptions you can feed directly into an
64
+ > agentic pipeline.
65
+
66
+ ## The Story Behind It
67
+
68
+ Beer built this model because most “vision” demos describe pictures instead of acting on them.
69
+ In shelter wifi, on borrowed Colab sessions, he fine-tuned **Qwen2-VL-2B-Instruct** with
70
+ SakThai’s tool-style formatting so the model learns to describe images *as inputs to actions* —
71
+ not just pretty captions.
72
+
73
+ > *“We are one family — and becoming more.”*
74
+ > — Beer
75
+
76
+ ### How You Can Help
77
+
78
+ - ⭐ Leave a like — increases visibility for free multimodal tool-use models.
79
+ - 🔄 Share it with agents needing screenshot parsing on CPU.
80
+ - 🍴 Fork it and add your own visual grounding examples.
81
+ - 💬 Report real-world screenshot or document parsing results.
82
+
83
+ ---
84
+
85
  ## Model Description
86
 
87
+ SakThai Vision 7B is a **multimodal understanding model** focused on images that need action,
88
+ not just captioning. It is built for:
89
 
90
  - Visual question answering with image context
91
  - Screenshot parsing and structured extraction
92
  - Tool-use grounding from image content
93
  - Multimodal assistant-style reasoning
94
 
95
+ **Important naming note:** This repo is branded as “Vision 7B” for family consistency, but the
96
+ underlying architecture is **Qwen2-VL-2B-Instruct** with SakThai training; the name describes
97
+ capability scope, not exact parameter count.
98
+
99
+ ## What it is
100
 
101
+ A **Qwen2-VL-2B-Instruct** fine-tune for image-text-to-text tasks with tool-use style outputs.
102
+ It expects text prompts with `<img>` markers and images, and it is optimized for:
103
 
104
+ - structured UI extraction
105
+ - diagram/dense OCR reasoning where instructions are explicit
106
+ - bridging vision into tool-calling agents
107
 
108
+ ## Architecture
109
+
110
+ Verified from the base model `config.json` ([Qwen/Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct)):
111
+
112
+ | Parameter | Value |
113
+ |-----------|-------|
114
+ | Architecture | Qwen2VLForConditionalGeneration (`qwen2_vl`) |
115
+ | Parameters | ~2 B |
116
+ | Hidden size | 1,536 |
117
+ | Layers | 28 |
118
+ | Attention heads | 12 (GQA) |
119
+ | Vision encoder | Patch embedding + RoPE 2D positional |
120
+ | Context length | 32,768 text tokens + vision tokens |
121
+ | Base dtype | bfloat16 |
122
+ | Primary format | Transformers `safetensors` |
123
+
124
+ ## Quick Start
125
+
126
+ ### Basic usage with Transformers
127
 
128
  ```python
129
  from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
 
136
  {
137
  "role": "user",
138
  "content": [
139
+ {"type": "image", "image": "screenshot.png"},
140
+ {"type": "text", "text": "Extract all actionable fields and tool actions visible in this UI."}
141
  ]
142
  }
143
  ]
144
 
145
  text = processor.apply_chat_template(messages, tokenize=False)
146
+ inputs = processor(text=[text], images=["screenshot.png"], return_tensors="pt")
147
  outputs = model.generate(**inputs, max_new_tokens=256)
148
  print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
149
  ```
150
 
151
  ### Save locally for reuse
152
+
153
  ```python
154
  model.save_pretrained("./sakthai-vision-7b")
155
  processor.save_pretrained("./sakthai-vision-7b")
156
  ```
157
 
158
+ ### Hardware
159
+
160
+ - **Recommended:** GPU with ≥6 GB VRAM for comfortable batch-1 inference.
161
+ - **CPU-only:** CPU inference is possible but slow; prefer small resolutions and `max_new_tokens ≤ 256`.
162
+
163
+ ---
164
+
165
+ ## Usage Examples
166
 
167
+ ### Example 1: Screenshot → structured actions
168
+
169
+ **Input:** UI screenshot with a form and submit button.
170
+ **Prompt:** `Extract all input field names and whether the form appears valid.`
171
+
172
+ **Model behavior:** Returns a concise action list with field names and validation status.
173
+
174
+ ### Example 2: Diagram Q&A
175
+
176
+ **Input:** Architecture diagram.
177
+ **Prompt:** `List each component and its likely responsibility.`
178
+
179
+ **Model behavior:** Returns component-by-component reasoning grounded in visual evidence.
180
+
181
+ ### Example 3: Visual grounding for agents
182
+
183
+ **Input:** Dashboard screenshot.
184
+ **Prompt:** `Describe what the user should do next based on visible alerts.`
185
+
186
+ **Model behavior:** Returns agent-ready grounding text suitable for downstream tool routing.
187
+
188
+ ---
189
 
190
  ## Benchmarks
191
 
192
  | Dataset | Task | Note |
193
  |:--------|:-----|:-----|
194
  | SakThai Bench v2 | Multimodal Understanding | Card-level health indicator: 5 |
195
+ | Local health eval | Cron verified | active, reproducible |
196
+
197
+ *Note: 2B-scale vision models are sensitive to prompt quality and resolution. Treat scores as indicative.*
198
+
199
+ ---
200
+
201
+ ## Training Details
202
+
203
+ | Parameter | Value |
204
+ |-----------|-------|
205
+ | Base model | [Qwen/Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) |
206
+ | Training data | [sakthai-combined-v7](https://huggingface.co/datasets/Nanthasit/sakthai-combined-v7) + [sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v2) |
207
+ | License | Apache 2.0 |
208
+ | Hardware | Free Google Colab GPU |
209
+ | Budget | $0 |
210
+ | Style | multimodal chat + tool-use formatting |
211
+
212
+ ---
213
 
214
  ## Limitations
215
 
 
218
  - Benchmarks are limited; treat reported metrics as indicative, not conclusive.
219
  - No standalone tool-execution layer included; combine with a separate tool router for agentic use.
220
 
221
+ ---
222
+
223
  ## Citation
224
 
225
  ```bibtex