Action100M: A Large-scale Video Action Dataset
Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling.
6969b18332f0333869ff9484
Delong Chen
claimed_verified
2026-01-16T14:37:03.915Z
630491107424d937fa3258be
/avatars/b8bd81bc8544674ee26b78702afdb87c.svg
Delong Chen
chendelong
user
chendelong
6969b18332f0333869ff9485
Tejaswi Kasarla
admin_assigned
2026-01-16T15:52:21.274Z
67a4de92a41dcedc1632eb43
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/rKpXABi8RqSeRh2RXBlZU.png
Tejaswi Kasarla
tkasarla
user
tkasarla
6969b18332f0333869ff9486
Yejin Bang
claimed_verified
2026-01-21T09:25:09.737Z
6566fd7447b037bd21f4147e
/avatars/c0da0ace33c6c163ec5fa5a866e1296b.svg
Yejin Bang
yjbang
user
yjbang
6969b18332f0333869ff9487
Mustafa Shukor
admin_assigned
2026-01-16T15:52:30.924Z
62bdeedd01dc22b4d22a371e
https://cdn-avatars.huggingface.co/v1/production/uploads/62bdeedd01dc22b4d22a371e/ahbK9Ehurx1TgQAVw1TcS.jpeg
Mustafa Shukor
mshukor
user
mshukor
6969b18332f0333869ff9488
Willy Chung
admin_assigned
2026-01-16T15:52:37.777Z
681abde923dbf6b59c410214
/avatars/f863d0a3d05fc99b29535d797d7a6292.svg
willy Chung
willy1027
user
willy1027
6969b18332f0333869ff9489
Jade Yu
6969b18332f0333869ff948a
Allen Bolourchi
admin_assigned
2026-01-16T15:52:48.723Z
65f08f0bcd30f1ed6ea6cbd1
/avatars/5eb2c2032dc2054d4ca86665ba0d428c.svg
Allen Bolourchi
allen-ml
user
allen-ml
6969b18332f0333869ff948b
Theo Moutakanni
admin_assigned
2026-01-16T15:52:55.020Z
639111acb73a0f8c02a92844
/avatars/58766701471be66fd784bedd319741a4.svg
Theo Moutakanni
TheoM
user
TheoM
6969b18332f0333869ff948c
Pascale Fung
Action100M: A Large-scale Video Action Dataset
Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling.
Action100M is a large-scale video action dataset constructed from internet instructional videos using automated pipelines with V-JEPA embeddings and GPT-based reasoning for structured annotations.
Action100M V-JEPA Tree-of-Captions GPT-OSS-120B Self-Refine video action recognition zero-shot performance data-scaling improvements