Ebisu: Benchmarking Large Language Models in Japanese Finance
Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.
69816fa4ce18b18628095fe0
Xueqing Peng
claimed_verified
2026-02-03T10:07:46.015Z
63a0c0803c8841cfe2cd1f15
/avatars/bbe216db7a33612f23d23ce4ed4ba3f9.svg
Xueqing Peng
Xueqing
user
Xueqing
69816fa4ce18b18628095fe1
Ruoyu Xiang
69816fa4ce18b18628095fe2
Fan Zhang
69816fa4ce18b18628095fe3
Mingzi Song
69816fa4ce18b18628095fe4
Mingyang Jiang
69816fa4ce18b18628095fe5
Yan Wang
69816fa4ce18b18628095fe6
Lingfei Qian
69816fa4ce18b18628095fe7
Taiki Hara
69816fa4ce18b18628095fe8
Yuqing Guo
69816fa4ce18b18628095fe9
Jimin Huang
69816fa4ce18b18628095fea
Junichi Tsujii
69816fa4ce18b18628095feb
Sophia Ananiadou
Ebisu: Benchmarking Large Language Models in Japanese Finance
Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.
A Japanese financial language understanding benchmark named Ebisu is introduced, featuring two expert-annotated tasks that evaluate implicit commitment recognition and hierarchical financial terminology extraction, revealing persistent challenges for current language models despite their advanced capabilities.
implicit commitment refusal recognition investor-facing Q&A hierarchical extraction nested financial terminology professional disclosures language model evaluation linguistic structure cultural norms high-context communication