Image-to-Text
Transformers
Safetensors
mistral3
text-generation
ocr
document-understanding
vision-language
pdf
tables
forms
Eval Results
🇪🇺 Region: EU
Instructions to use lightonai/LightOnOCR-1B-1025 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lightonai/LightOnOCR-1B-1025 with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="lightonai/LightOnOCR-1B-1025")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForSeq2SeqLM processor = AutoProcessor.from_pretrained("lightonai/LightOnOCR-1B-1025") model = AutoModelForSeq2SeqLM.from_pretrained("lightonai/LightOnOCR-1B-1025", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update tokenizer_config.json
Browse files- tokenizer_config.json +3 -0
tokenizer_config.json
CHANGED
|
@@ -231,6 +231,9 @@
|
|
| 231 |
"eos_token": "<|im_end|>",
|
| 232 |
"errors": "replace",
|
| 233 |
"extra_special_tokens": {},
|
|
|
|
|
|
|
|
|
|
| 234 |
"model_max_length": 131072,
|
| 235 |
"pad_token": "<|endoftext|>",
|
| 236 |
"processor_class": "LightOnOCRProcessor",
|
|
|
|
| 231 |
"eos_token": "<|im_end|>",
|
| 232 |
"errors": "replace",
|
| 233 |
"extra_special_tokens": {},
|
| 234 |
+
"image_token": "<|image_pad|>",
|
| 235 |
+
"image_break_token": "<|vision_pad|>",
|
| 236 |
+
"image_end_token": "<|vision_end|>",
|
| 237 |
"model_max_length": 131072,
|
| 238 |
"pad_token": "<|endoftext|>",
|
| 239 |
"processor_class": "LightOnOCRProcessor",
|