VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.
69ba52f65d2b9c28ea09f29a
Udi Barzelay
claimed_verified
2026-03-18T08:35:39.492Z
66323035d9fb78439d3a3b80
/avatars/04754ec424869b5f543260a8b2f36acf.svg
Barzelay
Udibarzi
user
Udibarzi
69ba52f65d2b9c28ea09f29b
Ophir Azulai
69ba52f65d2b9c28ea09f29c
Inbar Shapira
admin_assigned
2026-03-18T10:18:22.478Z
65d4d500493aeaf0bb2916f5
/avatars/14d698852fa8e345d0ce770a58759088.svg
Inbar Shapira
Inbars
user
Inbars
69ba52f65d2b9c28ea09f29d
Idan Friedman
claimed_verified
2026-03-19T13:01:48.577Z
66efd1134ed3540120d128d5
/avatars/038ff620903ee3dbcee565433ccc8aa1.svg
idan friedman
idanfriedman
user
idanfriedman
69ba52f65d2b9c28ea09f29e
Foad Abo Dahood
69ba52f65d2b9c28ea09f29f
Madison Lee
69ba52f65d2b9c28ea09f2a0
Abraham Daniels
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.
VAREX is a multimodal benchmark for structured data extraction from government forms that uses synthetic PDF templates and evaluates models across four input modalities to assess the impact of input format on extraction accuracy.
multimodal foundation models structured data extraction government forms Reverse Annotation pipeline PDF templates synthetic values ground truth quality assurance schema compliance schema echo extraction-specific fine-tuning layout-preserving text document image input modalities accuracy band