Instructions to use hanji-dev/hanji-parse-4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hanji-dev/hanji-parse-4b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hanji-dev/hanji-parse-4b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hanji-dev/hanji-parse-4b")
model = AutoModelForMultimodalLM.from_pretrained("hanji-dev/hanji-parse-4b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hanji-dev/hanji-parse-4b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hanji-dev/hanji-parse-4b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hanji-dev/hanji-parse-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hanji-dev/hanji-parse-4b

SGLang

How to use hanji-dev/hanji-parse-4b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hanji-dev/hanji-parse-4b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hanji-dev/hanji-parse-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hanji-dev/hanji-parse-4b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hanji-dev/hanji-parse-4b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hanji-dev/hanji-parse-4b with Docker Model Runner:
```
docker model run hf.co/hanji-dev/hanji-parse-4b
```

soamikapadia commited on 9 days ago

Commit

c0ad7ce

verified ·

1 Parent(s): b58d44a

Trim model card

Browse files

Files changed (1) hide show

README.md +6 -42

README.md CHANGED Viewed

@@ -22,10 +22,6 @@ content, a key-value panel, or a whole table) with a bounding box and its
 transcribed text. Tables are transcribed as GitHub-Flavored Markdown inside a
 single block. Figures, photos, and signatures are returned as image blocks.
-It was the production parsing model behind the Hanji document-extraction API.
-The accompanying open-source parse + schema-extraction server is available at
-**https://github.com/youlearn-ai/hanji**.
 ## Output contract
 The model returns **JSON only**: an array of records
@@ -48,10 +44,6 @@ The model returns **JSON only**: an array of records
 ## Usage — read this before running the model
-The model is **tightly coupled to its serving contract**. It was trained and
-served with one exact prompt, one image-preprocessing rule, and greedy
-decoding. Deviating from any of these produces off-distribution output.
 ### 1. Image preprocessing
 - Downscale so the image is at most **2,000,000 pixels** (2 MP), preserving
@@ -72,7 +64,7 @@ def preprocess(img: Image.Image) -> Image.Image:
     return img.convert("RGB").resize((w, h), Image.LANCZOS)
 ```
-### 2. The prompt (use it verbatim)
 Send the page image followed by exactly this text as the user turn. Do not
 paraphrase, extend, or reformat it.
@@ -202,7 +194,7 @@ model = AutoModelForImageTextToText.from_pretrained(
 processor = AutoProcessor.from_pretrained(MODEL)
 image = preprocess(Image.open("page.png"))  # see preprocessing above
-PROMPT = "..."  # the exact prompt above, verbatim
 messages = [{
     "role": "user",
@@ -220,11 +212,10 @@ out = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
 print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
-### Serving (SGLang — the production configuration)
-The model was served with SGLang `v0.5.12.post1` on a single H100. N-gram
-speculative decoding gives a large decode speedup on this output format (the
-repeated JSON keys draft extremely well):
 ```bash
 python -m sglang.launch_server \
@@ -241,36 +232,9 @@ python -m sglang.launch_server \
 ```
 A ready-to-run server that implements the full preprocessing + prompt contract
-(and a Baseten deployment config) ships with the open-source repo:
 https://github.com/youlearn-ai/hanji.
-## Training summary
-- Base: `Qwen/Qwen3-VL-4B-Instruct` (Apache-2.0). Language layers fine-tuned
-  with LoRA (r=32, α=64, all attention + MLP projections); the **vision tower
-  was frozen**. This repository contains the merged full weights (bf16).
-- 1,500 steps, effective batch 16, cosine schedule, images at up to 2 MP
-  native resolution.
-- ~16k supervised pages spanning synthetic and real business documents:
-  forms, tables (including borderless and dense grids), checkbox and Y/N
-  grids, multilingual and RTL pages, degraded scans and faxes, receipts, and
-  court transcripts.
-## Limitations
-- **Prompt-coupled**: output quality degrades sharply off the exact prompt,
-  preprocessing, and greedy decoding described above.
-- The base model was loaded from the `main` revision of
-  `Qwen/Qwen3-VL-4B-Instruct` at training time (mid-2026) without a pinned
-  commit; bit-exact re-derivation of the merge is therefore approximate.
-- Signatures and handwritten marks are deliberately returned as `<image>`
-  blocks, never transcribed.
-- Blocks are page-scoped; the model does not merge tables that continue
-  across pages (the serving pipeline may do so downstream).
-- Not an instruction-following chat model: it does one task. For arbitrary
-  document Q&A, pair it with a downstream LLM (the open-source repo pairs it
-  with a schema-extraction pass).
 ## License
 Apache-2.0. Fine-tuned from

 transcribed text. Tables are transcribed as GitHub-Flavored Markdown inside a
 single block. Figures, photos, and signatures are returned as image blocks.
 ## Output contract
 The model returns **JSON only**: an array of records
 ## Usage — read this before running the model
 ### 1. Image preprocessing
 - Downscale so the image is at most **2,000,000 pixels** (2 MP), preserving
     return img.convert("RGB").resize((w, h), Image.LANCZOS)
 ```
+### 2. The prompt
 Send the page image followed by exactly this text as the user turn. Do not
 paraphrase, extend, or reformat it.
 processor = AutoProcessor.from_pretrained(MODEL)
 image = preprocess(Image.open("page.png"))  # see preprocessing above
+PROMPT = "..."  # the exact prompt above
 messages = [{
     "role": "user",
 print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
+### Serving (SGLang)
+N-gram speculative decoding gives a large decode speedup on this output
+format (the repeated JSON keys draft extremely well):
 ```bash
 python -m sglang.launch_server \
 ```
 A ready-to-run server that implements the full preprocessing + prompt contract
+(and a schema-extraction API around it) is available at
 https://github.com/youlearn-ai/hanji.
 ## License
 Apache-2.0. Fine-tuned from