BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic
  Domains
The rapid advancement of large language models(LLMs) has intensified the need
for domain and culture specific evaluation. Existing benchmarks are largely
Anglocentric and domain-agnostic, limiting their applicability to India-centric
contexts. To address this gap, we introduce BhashaBench V1, the first
domain-specific, multi-task, bilingual benchmark focusing on critical Indic
knowledge systems. BhashaBench V1 contains 74,166 meticulously curated
question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from
authentic government and domain-specific exams. It spans four major domains:
Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and
covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs
reveals significant domain and language specific performance gaps, with
especially large disparities in low-resource domains. For instance, GPT-4o
achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models
consistently perform better on English content compared to Hindi across all
domains. Subdomain-level analysis shows that areas such as Cyber Law,
International Finance perform relatively well, while Panchakarma, Seed Science,
and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive
dataset for evaluating large language models across India's diverse knowledge
domains. It enables assessment of models' ability to integrate domain-specific
knowledge with bilingual understanding. All code, benchmarks, and resources are
publicly available to support open research.
6902c31d72739622ee92a68d
Vijay Devane
claimed_verified
2025-10-30T14:32:34.597Z
64d22cc68f912d78587f396f
https://cdn-avatars.huggingface.co/v1/production/uploads/64d22cc68f912d78587f396f/g50XH8IbGu5uG3X02CDrM.jpeg
Vijay Devane
vjdevane
user
vjdevane
6902c31d72739622ee92a68e
Mohd Nauman
claimed_verified
2025-11-13T13:07:26.482Z
67f386acaa1072e93b0f1823
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hj9-bM3F3sDRhMEu0wZKV.png
Mohd Nauman
nauman-tih
user
nauman-tih
6902c31d72739622ee92a68f
Bhargav Patel
6902c31d72739622ee92a690
Aniket Mahendra Wakchoure
6902c31d72739622ee92a691
Yogeshkumar Sant
6902c31d72739622ee92a692
Shyam Pawar
6902c31d72739622ee92a693
Viraj Thakur
6902c31d72739622ee92a694
Ananya Godse
6902c31d72739622ee92a695
Sunil Patra
6902c31d72739622ee92a696
Neha Maurya
6902c31d72739622ee92a697
Suraj Racha
6902c31d72739622ee92a698
Nitish Kamal Singh
6902c31d72739622ee92a699
Ajay Nagpal
6902c31d72739622ee92a69a
Piyush Sawarkar
6902c31d72739622ee92a69b
Kundeshwar Vijayrao Pundalik
6902c31d72739622ee92a69c
Rohit Saluja
6902c31d72739622ee92a69d
Ganesh Ramakrishnan
BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic
  Domains
The rapid advancement of large language models(LLMs) has intensified the need
for domain and culture specific evaluation. Existing benchmarks are largely
Anglocentric and domain-agnostic, limiting their applicability to India-centric
contexts. To address this gap, we introduce BhashaBench V1, the first
domain-specific, multi-task, bilingual benchmark focusing on critical Indic
knowledge systems. BhashaBench V1 contains 74,166 meticulously curated
question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from
authentic government and domain-specific exams. It spans four major domains:
Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and
covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs
reveals significant domain and language specific performance gaps, with
especially large disparities in low-resource domains. For instance, GPT-4o
achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models
consistently perform better on English content compared to Hindi across all
domains. Subdomain-level analysis shows that areas such as Cyber Law,
International Finance perform relatively well, while Panchakarma, Seed Science,
and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive
dataset for evaluating large language models across India's diverse knowledge
domains. It enables assessment of models' ability to integrate domain-specific
knowledge with bilingual understanding. All code, benchmarks, and resources are
publicly available to support open research.
BhashaBench V1 is a domain-specific, multi-task, bilingual benchmark for evaluating large language models in critical Indic knowledge systems, revealing significant performance gaps across domains and languages.
large language models domain-specific multi-task bilingual benchmark Indic knowledge systems question-answer pairs government exams domain-specific exams Agriculture Legal Finance Ayurveda subdomains topics evaluation performance gaps low-resource domains GPT-4 English Hindi Cyber Law International Finance Panchakarma Seed Science Human Rights