FinForge: Semi-Synthetic Financial Benchmark Generation
Evaluating Language Models (LMs) in specialized, high-stakes domains such as finance remains a significant challenge due to the scarcity of open, high-quality, and domain-specific datasets. Existing general-purpose benchmarks provide broad coverage but lack the depth and domain fidelity needed to assess LMs' capabilities for real-world financial reasoning, which requires both conceptual understanding and quantitative rigor. To address this gap, we introduce FinForge, a scalable, semi-synthetic pipeline for constructing finance-specific evaluation benchmarks through a hybrid of expert-guided data curation and controlled LM-based synthesis. FinForge combines manual and programmatic corpus construction from authoritative financial sources with structured question generation and validation using Gemini 2.5 Flash. To demonstrate the pipeline's efficacy, we produce FinForge-5k, a snapshot benchmark comprising over 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from a curated corpus of 100,000 verified documents totaling 143M tokens. Evaluation of state-of-the-art open-source and closed-source models on FinForge-5k reveals significant differences in financial reasoning, with leading models achieving accuracy levels near 80%. These findings underscore the framework's utility for diagnosing current model limitations and guiding future improvements in financial domain competence. All code and data are available at https://github.com/gtfintechlab/FinForge.
6965de3ffc8c4ecc02c7fa1d
Glenn Matlin
6965de3ffc8c4ecc02c7fa1e
Akhil Theerthala
claimed_verified
2026-01-13T08:23:32.745Z
676c67da7bad1587f2d046e2
https://cdn-avatars.huggingface.co/v1/production/uploads/676c67da7bad1587f2d046e2/RAqRxiaEHpB6qFCQ-11RL.jpeg
Akhil Theerthala
Akhil-Theerthala
user
Akhil-Theerthala
6965de3ffc8c4ecc02c7fa1f
Anant Gupta
claimed_verified
2026-04-16T08:22:12.047Z
6636d57fef943e62957c0b00
/avatars/b5ade2cfd3c80ca5f86246103bbbecac.svg
Anant
anantgupta2
user
anantgupta2
6965de3ffc8c4ecc02c7fa20
Anirudh JM
6965de3ffc8c4ecc02c7fa21
Rayan Castilla
6965de3ffc8c4ecc02c7fa22
Yi Mei Ng
6965de3ffc8c4ecc02c7fa23
Sudheer Chava
FinForge: Semi-Synthetic Financial Benchmark Generation
Evaluating Language Models (LMs) in specialized, high-stakes domains such as finance remains a significant challenge due to the scarcity of open, high-quality, and domain-specific datasets. Existing general-purpose benchmarks provide broad coverage but lack the depth and domain fidelity needed to assess LMs' capabilities for real-world financial reasoning, which requires both conceptual understanding and quantitative rigor. To address this gap, we introduce FinForge, a scalable, semi-synthetic pipeline for constructing finance-specific evaluation benchmarks through a hybrid of expert-guided data curation and controlled LM-based synthesis. FinForge combines manual and programmatic corpus construction from authoritative financial sources with structured question generation and validation using Gemini 2.5 Flash. To demonstrate the pipeline's efficacy, we produce FinForge-5k, a snapshot benchmark comprising over 5,000 human-validated question-answer pairs across 11 finance subdomains, derived from a curated corpus of 100,000 verified documents totaling 143M tokens. Evaluation of state-of-the-art open-source and closed-source models on FinForge-5k reveals significant differences in financial reasoning, with leading models achieving accuracy levels near 80%. These findings underscore the framework's utility for diagnosing current model limitations and guiding future improvements in financial domain competence. All code and data are available at https://github.com/gtfintechlab/FinForge.
FinForge presents a scalable semi-synthetic pipeline for creating domain-specific financial evaluation benchmarks using expert curation and language model synthesis, demonstrating significant variations in financial reasoning capabilities among state-of-the-art models.
language models finance-specific evaluation benchmarks semi-synthetic pipeline expert-guided data curation controlled LM-based synthesis question-generation validation FinForge-5k financial reasoning domain competence