Multimodal OCR: Parse Anything from Documents
We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, dots.mocr achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.
69b7798b91a28162b233fc66
Handong Zheng
69b7798b91a28162b233fc67
Yumeng Li
69b7798b91a28162b233fc68
Kaile Zhang
69b7798b91a28162b233fc69
Liang Xin
69b7798b91a28162b233fc6a
Guangwei Zhao
69b7798b91a28162b233fc6b
Hao Liu
69b7798b91a28162b233fc6c
Jiayu Chen
69b7798b91a28162b233fc6d
Jie Lou
69b7798b91a28162b233fc6e
Jiyu Qiu
69b7798b91a28162b233fc6f
Qi Fu
69b7798b91a28162b233fc70
Rui Yang
69b7798b91a28162b233fc71
Shuo Jiang
69b7798b91a28162b233fc72
Weijian Luo
69b7798b91a28162b233fc73
Weijie Su
69b7798b91a28162b233fc74
Weijun Zhang
69b7798b91a28162b233fc75
Xingyu Zhu
69b7798b91a28162b233fc76
Yabin Li
69b7798b91a28162b233fc77
Yiwei ma
69b7798b91a28162b233fc78
Yu Chen
69b7798b91a28162b233fc79
Zhaohui Yu
admin_assigned
2026-03-16T11:03:29.542Z
641bd4511911d3be6747ba88
/avatars/405785b30411dd76c0e7db9bab86c684.svg
zhaohuiyu
Zhaohuiyu
user
Zhaohuiyu
69b7798b91a28162b233fc7a
Guang Yang
69b7798b91a28162b233fc7b
Colin Zhang
69b7798b91a28162b233fc7c
Lei Zhang
69b7798b91a28162b233fc7d
Yuliang Liu
69b7798b91a28162b233fc7e
Xiang Bai
Multimodal OCR: Parse Anything from Documents
We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, dots.mocr achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.
MOCR is a multimodal OCR approach that jointly parses text and graphics into unified representations, enabling structured document reconstruction and supporting end-to-end training with semantic relations between visual and textual components.
document parsing multimodal OCR structured outputs end-to-end training semantic relations staged pretraining supervised fine-tuning image-to-code corpora