Instructions to use aliRafik/invoices-donut-finetuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aliRafik/invoices-donut-finetuned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="aliRafik/invoices-donut-finetuned")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("aliRafik/invoices-donut-finetuned")
model = AutoModelForImageTextToText.from_pretrained("aliRafik/invoices-donut-finetuned")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use aliRafik/invoices-donut-finetuned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aliRafik/invoices-donut-finetuned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aliRafik/invoices-donut-finetuned",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/aliRafik/invoices-donut-finetuned

SGLang

How to use aliRafik/invoices-donut-finetuned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aliRafik/invoices-donut-finetuned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aliRafik/invoices-donut-finetuned",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aliRafik/invoices-donut-finetuned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aliRafik/invoices-donut-finetuned",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use aliRafik/invoices-donut-finetuned with Docker Model Runner:
```
docker model run hf.co/aliRafik/invoices-donut-finetuned
```

Model Card for invoices-donut-finetuned

This model is a fine-tuned version of numind/NuExtract-2.0-4B. It has been trained using TRL.

Quick start

from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="aliRafik/invoices-donut-finetuned", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training procedure

This model was trained with SFT.

Framework versions

TRL: 0.21.0
Transformers: 4.56.0.dev0
Pytorch: 2.5.1+cu121
Datasets: 4.0.0
Tokenizers: 0.21.4

invoices-donut-finetuned-full

Overview

invoices-donut-finetuned-full is a fully fine-tuned version of numind/NuExtract-2.0-4B, itself built on Qwen/Qwen2.5-VL-3B-Instruct. It has been trained on the katanaml-org/invoices-donut-data-v1 dataset (501 invoice samples). The model extracts structured invoice information, including headers, itemized products, and financial summaries.

Intended Use

Extracting structured JSON fields from invoice images:
- Invoice number, date
- Seller/client details
- Tax IDs, IBAN
- Item descriptions, prices, VAT
- Totals (net, VAT, gross)
Not intended for general document OCR outside invoices.

Training Details

Base model: Qwen/Qwen2.5-VL-3B-Instruct
Framework: Hugging Face TRL (SFTTrainer)
Epochs: 15
Batch size: 1
Learning rate: 1e-5
Precision: bfloat16
Gradient accumulation: 4
Scheduler: Constant LR
Max sequence length: 1024
Gradient checkpointing: Enabled
All weights were updated (no adapters).

Usage

Installation

pip install transformers torch datasets pillow

Load Model and Processor

import torch
from transformers import AutoProcessor, AutoModelForVision2Seq

model_name = "aliRafik/invoices-donut-finetuned"  

model = AutoModelForVision2Seq.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,  # Optional: Use float32 if bfloat16 causes issues
    attn_implementation="flash_attention_2",  # Requires Ampere+ GPU & torch >= 2.0
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    model_name,
    trust_remote_code=True,
    padding_side='left',
    use_fast=True
)

Alternative Loading for T4 GPUs (using SDPA)

model = AutoModelForVision2Seq.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,  # T4 supports bfloat16
    attn_implementation="sdpa",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    model_name,
    trust_remote_code=True,
    padding_side='left',
    use_fast=True
)

Define Extraction Template

template = """
{
  "header": {
    "invoice_no": "string",
    "invoice_date": "date-time",
    "seller": "string",
    "client": "string",
    "seller_tax_id": "string",
    "client_tax_id": "string",
    "iban": "string"
  },
  "items": [
    {
      "item_desc": "string",
      "item_qty": "number",
      "item_net_price": "number",
      "item_net_worth": "number",
      "item_vat": "number",
      "item_gross_worth": "number"
    }
  ],
  "summary": {
    "total_net_worth": "number",
    "total_vat": "number",
    "total_gross_worth": "number"
  }
}
"""

Test on Sample from Dataset

from datasets import load_dataset
import json
from qwen_vl_utils import process_vision_info

# Load the dataset
dataset = load_dataset("katanaml-org/invoices-donut-data-v1")

# Select a sample (e.g., index 0)
sample = dataset['train'][0]
image = sample['image']
ground_truth = sample['ground_truth']

print(json.loads(ground_truth))

# Prepare message
messages = [
    {"role": "user", "content": [{"type": "image", "image": image}]}
]

# Process vision info
image_inputs, _ = process_vision_info(messages)

# Apply chat template
text = processor.tokenizer.apply_chat_template(
    messages,
    template=template,
    tokenize=False,
    add_generation_prompt=True
)

# Prepare inputs
inputs = processor(
    text=[text],
    images=image_inputs,
    padding=True,
    return_tensors="pt"
).to(model.device)

# Generation config
generation_config = {
    "do_sample": False,
    "num_beams": 1,
    "max_new_tokens": 2048
}

# Generate
generated_ids = model.generate(**inputs, **generation_config)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

# Parse and print
try:
    extracted_data = json.loads(output_text[0])
    print("Extracted Data:", extracted_data)
except json.JSONDecodeError:
    print("Raw Output:", output_text[0])

# Compare with ground truth
gt_parsed = json.loads(ground_truth)['gt_parse']
print("Ground Truth:", gt_parsed)

Test on Unseen Data (Custom Image)

from PIL import Image
from io import BytesIO
import requests

# Load from local path
image_path = "/content/image.jpg"  # Replace with your path
image = Image.open(image_path)

# Or load from URL
# image_url = "https://example.com/your_invoice.jpg"
# response = requests.get(image_url)
# image = Image.open(BytesIO(response.content))

# Use same inference code as above

Example Results

Input Image:

Extracted Data:


{
  "header": {
    "invoice_no": "49565075",
    "invoice_date": "2019-10-28",
    "seller": "Kane-Morgan 968 Carr Mission Apt. 320 Bernardville, VA 28211",
    "client": "Garcia Inc 445 Haas Viaduct Suite 454 Michaelhaven, LA 32852",
    "seller_tax_id": "964-95-3813",
    "client_tax_id": "909-75-5482",
    "iban": "GB73WCJ55232646970614"
  },
  "items": [
    {
      "item_desc": "Anthropologie Gold Elegant Swan Decorative Metal Bottle Stopper Wine Saver",
      "item_qty": 3.0,
      "item_net_price": 19.98,
      "item_net_worth": 59.94,
      "item_vat": 10.0,
      "item_gross_worth": 65.93
    },
    {
      "item_desc": "Lolita Happy Retirement Wine Glass 15 Ounce GLS11-5534H",
      "item_qty": 1.0,
      "item_net_price": 8.0,
      "item_net_worth": 8.0,
      "item_vat": 10.0,
      "item_gross_worth": 8.8
    },
    {
      "item_desc": "Lolita \"Congratulations\" Hand Painted and Decorated Wine Glass NIB",
      "item_qty": 1.0,
      "item_net_price": 20.0,
      "item_net_worth": 20.0,
      "item_vat": 10.0,
      "item_gross_worth": 22.0
    }
  ],
  "summary": {
    "total_net_worth": 87.94,
    "total_vat": 8.79,
    "total_gross_worth": 96.73
  }
}

License

Apache-2.0

tags:

vision

document-understanding

invoice-processing

donut

qwen

Citations

Cite TRL as:

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}

Downloads last month: 64

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for aliRafik/invoices-donut-finetuned

Base model

Qwen/Qwen2.5-VL-3B-Instruct

Finetuned

numind/NuExtract-2.0-4B

Finetuned

(2)

this model