AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.
691d6caa0247f3d258ef32a6
Mohammad Zbib
claimed_verified
2025-11-20T08:49:50.964Z
665ee96b9f29909d03abfa37
/avatars/0de2329d23206d892a6deac8f27de7c5.svg
Mohamad Zbib
zbeeb
user
zbeeb
691d6caa0247f3d258ef32a7
Hasan Abed Al Kader Hammoud
claimed_verified
2025-11-20T08:49:52.684Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
hammh0a
user
hammh0a
691d6caa0247f3d258ef32a8
Sina Mukalled
691d6caa0247f3d258ef32a9
Nadine Rizk
claimed_verified
2025-11-21T09:02:12.659Z
691e2375c8017170adcebe08
/avatars/3baacc198bdad6d369a2a71cfb5ded9b.svg
Nadine Rizk
Nadine-rizk18
user
Nadine-rizk18
691d6caa0247f3d258ef32aa
Fatima Karnib
claimed_verified
2025-11-24T07:59:02.169Z
68f367a6d91682eeff678467
/avatars/edd68b9b1b03c7a84e34d8de8476d7b2.svg
fatima karnib
fatima123karnib
user
fatima123karnib
691d6caa0247f3d258ef32ab
Issam Lakkis
691d6caa0247f3d258ef32ac
Ammar Mohanna
claimed_verified
2025-11-19T08:39:11.874Z
61f01168943fd4875cd86724
/avatars/415cc5388c16e2527d286bc56faeb810.svg
Ammar Mohanna
AmmarMohanna
user
AmmarMohanna
691d6caa0247f3d258ef32ad
Bernard Ghanem
AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models
We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.
AraLingBench evaluates Arabic and bilingual LLMs' linguistic competence using a benchmark with expert-designed questions across grammar, morphology, spelling, reading comprehension, and syntax, revealing gaps between surface proficiency and true comprehension.
large language models LLMS grammar morphology spelling reading comprehension syntax multiple choice questions structural language understanding