FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
This paper introduces FinMCP-Bench, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.
69c5f0752de2ee20094edbfe
Jie Zhu
claimed_verified
2026-03-27T11:33:58.615Z
642656cbad1e3b0e6e91b752
/avatars/3bf0ee15fd528e09b2b889f5cce3cbd0.svg
Jie Zhu
amazingj
user
amazingj
69c5f0752de2ee20094edbff
Yimin Tian
69c5f0752de2ee20094edc00
Boyang Li
69c5f0752de2ee20094edc01
Kehao Wu
69c5f0752de2ee20094edc02
Zhongzhi Liang
69c5f0752de2ee20094edc03
Junhui Li
69c5f0752de2ee20094edc04
Xianyin Zhang
69c5f0752de2ee20094edc05
Lifan Guo
69c5f0752de2ee20094edc06
Feng Chen
69c5f0752de2ee20094edc07
Yong Liu
69c5f0752de2ee20094edc08
Chi Zhang
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
This paper introduces FinMCP-Bench, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.
FinMCP-Bench is a comprehensive benchmark for evaluating large language models on financial problem-solving through tool invocation and reasoning across multiple complexity levels.
large language models tool invocation financial model context protocols benchmark financial LLM agents