VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding
This work introduces VERSE, a methodology for analyzing and improving Vision-Language Models applied to Visually-rich Document Understanding by exploring their visual embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.
6960d5bd5b7998385e6396cc
Ignacio de Rodrigo
claimed_verified
2026-01-09T15:45:59.881Z
63ee535a190ddd6214f30dc2
https://cdn-avatars.huggingface.co/v1/production/uploads/63ee535a190ddd6214f30dc2/f4pNRFs9XVLToponWndWJ.jpeg
de Rodrigo
de-Rodrigo
user
de-Rodrigo
6960d5bd5b7998385e6396cd
Alvaro J. Lopez-Lopez
claimed_verified
2026-01-09T15:45:57.718Z
645c966858f9ee315142a2e5
/avatars/51928db15dddc07a51cc35a402f511c7.svg
Alvaro Lopez
alvlopez
user
alvlopez
6960d5bd5b7998385e6396ce
Jaime Boal
VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding
This work introduces VERSE, a methodology for analyzing and improving Vision-Language Models applied to Visually-rich Document Understanding by exploring their visual embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.
VERSE is a methodology for analyzing and improving Vision-Language Models in document understanding by visualizing latent representations and generating synthetic data to enhance performance in error-prone clusters.
Vision-Language Models visual embedding space latent representations synthetic data generation MERIT Dataset F1 performance Donut Idefics2 GPT-4 Pixtral