Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural
  Processing of Figurative Language
We present a comprehensive evaluation of the ability of large language models
(LLMs) to process culturally grounded language, specifically to understand and
pragmatically use figurative expressions that encode local knowledge and
cultural nuance. Using figurative language as a proxy for cultural nuance and
local knowledge, we design evaluation tasks for contextual understanding,
pragmatic use, and connotation interpretation in Arabic and English. We
evaluate 22 open- and closed-source LLMs on Egyptian Arabic idioms,
multidialectal Arabic proverbs, and English proverbs. Our results show a
consistent hierarchy: the average accuracy for Arabic proverbs is 4.29% lower
than for English proverbs, and performance for Egyptian idioms is 10.28% lower
than for Arabic proverbs. For the pragmatic use task, accuracy drops by 14.07%
relative to understanding, though providing contextual idiomatic sentences
improves accuracy by 10.66%. Models also struggle with connotative meaning,
reaching at most 85.58% agreement with human annotators on idioms with 100%
inter-annotator agreement. These findings demonstrate that figurative language
serves as an effective diagnostic for cultural reasoning: while LLMs can often
interpret figurative meaning, they face challenges in using it appropriately.
To support future research, we release Kinayat, the first dataset of Egyptian
Arabic idioms designed for both figurative understanding and pragmatic use
evaluation.
6902843d72739622ee92a618
Mena Attia
6902843d72739622ee92a619
Aashiq Muhamed
6902843d72739622ee92a61a
Mai Alkhamissi
6902843d72739622ee92a61b
Thamar Solorio
6902843d72739622ee92a61c
Mona Diab
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural
  Processing of Figurative Language
We present a comprehensive evaluation of the ability of large language models
(LLMs) to process culturally grounded language, specifically to understand and
pragmatically use figurative expressions that encode local knowledge and
cultural nuance. Using figurative language as a proxy for cultural nuance and
local knowledge, we design evaluation tasks for contextual understanding,
pragmatic use, and connotation interpretation in Arabic and English. We
evaluate 22 open- and closed-source LLMs on Egyptian Arabic idioms,
multidialectal Arabic proverbs, and English proverbs. Our results show a
consistent hierarchy: the average accuracy for Arabic proverbs is 4.29% lower
than for English proverbs, and performance for Egyptian idioms is 10.28% lower
than for Arabic proverbs. For the pragmatic use task, accuracy drops by 14.07%
relative to understanding, though providing contextual idiomatic sentences
improves accuracy by 10.66%. Models also struggle with connotative meaning,
reaching at most 85.58% agreement with human annotators on idioms with 100%
inter-annotator agreement. These findings demonstrate that figurative language
serves as an effective diagnostic for cultural reasoning: while LLMs can often
interpret figurative meaning, they face challenges in using it appropriately.
To support future research, we release Kinayat, the first dataset of Egyptian
Arabic idioms designed for both figurative understanding and pragmatic use
evaluation.
LLMs struggle with understanding and pragmatically using culturally grounded figurative language, particularly in Arabic, despite performing better with English proverbs.
large language models LLMs figurative expressions cultural nuance local knowledge evaluation tasks contextual understanding pragmatic use connotation interpretation Egyptian Arabic idioms multidialectal Arabic proverbs English proverbs Kinayat dataset