FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios
The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.
69d85b1879548974da9464bd
Xiangru Jian
claimed_verified
2026-04-10T13:34:13.501Z
636865b8cca0a0a962c21f3f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Mja7cpws4gb2Jmdj_foPA.png
Xiangru (Edward) Jian
HideOnBush
user
HideOnBush
69d85b1879548974da9464be
Hao Xu
claimed_verified
2026-04-13T19:24:31.304Z
66c9b8a03864174f35a51b28
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0tUoi-ddcDH6mgNLrfgEk.png
hao xu
moreerom
user
moreerom
69d85b1879548974da9464bf
Wei Pang
claimed_verified
2026-04-10T07:43:26.563Z
65164444bc0631719873af81
/avatars/dab8b90db8bbd00806268fe276e3ea36.svg
Wei Pang
weipang142857
user
weipang142857
69d85b1879548974da9464c0
Xinjian Zhao
claimed_verified
2026-04-13T07:33:58.220Z
68380f4f231cf484dd4e87f4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/xp34hfiSLf-DiE1DVVhHk.png
Xinjian Zhao
Xinjiansz
user
Xinjiansz
69d85b1879548974da9464c1
Chengyu Tao
69d85b1879548974da9464c2
Qixin Zhang
69d85b1879548974da9464c3
Xikun Zhang
69d85b1879548974da9464c4
Chao Zhang
69d85b1879548974da9464c5
Guanzhi Deng
69d85b1879548974da9464c6
Alex Xue
69d85b1879548974da9464c7
Juan Du
69d85b1879548974da9464c8
Tianshu Yu
69d85b1879548974da9464c9
Garth Tarr
69d85b1879548974da9464ca
Linqi Song
69d85b1879548974da9464cb
Qiuzhuang Sun
69d85b1879548974da9464cc
Dacheng Tao
FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios
The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.
FORGE introduces a high-quality multimodal manufacturing dataset with fine-grained domain semantics to evaluate MLLMs on real-world tasks, revealing that domain-specific knowledge rather than visual grounding limits performance, and demonstrating that supervised fine-tuning on structured annotations significantly improves accuracy.
Multimodal Large Language Models visual grounding domain-specific knowledge supervised fine-tuning 3B-parameter model fine-grained domain semantics workpiece verification structural surface inspection assembly verification