AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.
6923c67eb5612535ed95590c
Ren Ma
claimed_verified
2025-12-02T08:49:59.848Z
62581ef5d47cce537ee25963
/avatars/d987c625791dc33c84bf9b51069bf2c8.svg
Ren Ma
renma
user
renma
6923c67eb5612535ed95590d
Jiantao Qiu
claimed_verified
2025-12-05T08:30:21.771Z
641bbd19af42e9b7dd4638ce
/avatars/8e7819fa76650cab30e60063e8d3a217.svg
Qiu Jiantao
qiujiantao
user
qiujiantao
6923c67eb5612535ed95590e
Chao Xu
6923c67eb5612535ed95590f
Pei Chu
6923c67eb5612535ed955910
Kaiwen Liu
6923c67eb5612535ed955911
Pengli Ren
6923c67eb5612535ed955912
Yuan Qu
6923c67eb5612535ed955913
Jiahui Peng
6923c67eb5612535ed955914
Linfeng Hou
6923c67eb5612535ed955915
Mengjie Liu
claimed_verified
2026-01-21T10:16:05.637Z
672dbf1bec84b9c33412488f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V6o6K28ydpkaS9XFA4Ac3.png
Mengjie Liu
Balalauuoo
user
Balalauuoo
6923c67eb5612535ed955916
Lindong Lu
6923c67eb5612535ed955917
Wenchang Ning
6923c67eb5612535ed955918
Jia Yu
6923c67eb5612535ed955919
Rui Min
6923c67eb5612535ed95591a
Jin Shi
6923c67eb5612535ed95591b
Haojiong Chen
6923c67eb5612535ed95591c
Peng Zhang
6923c67eb5612535ed95591d
Wenjian Zhang
6923c67eb5612535ed95591e
Qian Jiang
6923c67eb5612535ed95591f
Zengjie Hu
claimed_verified
2025-11-26T09:28:21.069Z
66fe2be698f30194f8816e81
https://cdn-avatars.huggingface.co/v1/production/uploads/66fe2be698f30194f8816e81/RJgGWkn6Tc-EP7OvvmN91.jpeg
Zengjie Hu
FloSophorae
user
FloSophorae
6923c67eb5612535ed955920
Guoqiang Yang
6923c67eb5612535ed955921
Zhenxiang Li
6923c67eb5612535ed955922
Fukai Shang
6923c67eb5612535ed955923
Zhongying Tu
6923c67eb5612535ed955924
Wentao Zhang
6923c67eb5612535ed955925
Dahua Lin
6923c67eb5612535ed955926
Conghui He
AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction.
A novel extraction pipeline using a language model improves web data quality, significantly enhancing the performance of large language models trained on extracted corpora.
sequence labeling language model semantic understanding two-stage formatting pipeline Markdown ROUGE-N F1 structured element preservation multilingual corpus pretraining experiments benchmarks AI-ready Common Crawl RefinedWeb FineWeb HTML extraction