New foundation model on document understanding and generation in  transformers 🤩
text
New foundation model on document understanding and generation in  transformers 🤩
new_line
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
text
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
new_line
Demo 👉 
text
Demo 👉 
https://huggingface.co/spaces/merve/UDOP
merve/UDOP
space
resource
https://huggingface.co/spaces/merve/UDOP
new_line
It is a model that combines vision, text and layout. 📝
text
It is a model that combines vision, text and layout. 📝
new_line
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
text
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
new_line
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
text
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
new_line
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
text
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
new_line
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
text
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
new_line
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
text
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
new_line
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
text
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
new_line
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
text
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
new_line
You can learn more about the model from below resources (h/t to 
text
You can learn more about the model from below resources (h/t to 
new_line
@nielsr
mention
nielsr
), thanks a lot for reading 🤗
text
), thanks a lot for reading 🤗
new_line
Docs: 
text
Docs: 
https://huggingface.co/docs/transformers/main/en/model_doc/udop
https://huggingface.co/docs/transformers/main/en/model_doc/udop
link
 📚 
text
 📚 
new_line
Checkpoints: 
text
Checkpoints: 
https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
microsoft/udop-65e625124aee97415b88b513
collection
resource
https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
text
new_line
Demo notebooks: 
text
Demo notebooks: 
https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP
https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP
link
 📕
text
 📕
New foundation model on document understanding and generation in  transformers 🤩
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
Demo 👉 https://huggingface.co/spaces/merve/UDOP
It is a model that combines vision, text and layout. 📝
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
You can learn more about the model from below resources (h/t to 
@nielsr), thanks a lot for reading 🤗
Docs: https://huggingface.co/docs/transformers/main/en/model_doc/udop 📚 
Checkpoints: https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513 
Demo notebooks: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP 📕
❤️
samusenps rajistics not-lain cstr diwank