Probing Cultural Signals in Large Language Models through Author Profiling
Large language models (LLMs) are increasingly deployed in applications with societal impact, raising concerns about the cultural biases they encode. We probe these representations by evaluating whether LLMs can perform author profiling from song lyrics in a zero-shot setting, inferring singers' gender and ethnicity without task-specific fine-tuning. Across several open-source models evaluated on more than 10,000 lyrics, we find that LLMs achieve non-trivial profiling performance but demonstrate systematic cultural alignment: most models default toward North American ethnicity, while DeepSeek-1.5B aligns more strongly with Asian ethnicity. This finding emerges from both the models' prediction distributions and an analysis of their generated rationales. To quantify these disparities, we introduce two fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), and show that Ministral-8B displays the strongest ethnicity bias among the evaluated models, whereas Gemma-12B shows the most balanced behavior. Our code is available on GitHub (https://github.com/ValentinLafargue/CulturalProbingLLM).
69bbc54d081b9ad49af56e98
Valentin Lafargue
claimed_verified
2026-03-23T10:56:08.992Z
65436b74d4113e428f9fa93d
/avatars/0310eb707376991f50f9f42fc8a40b3d.svg
Valentin Lafargue
ValentinLAFARGUE
user
ValentinLAFARGUE
69bbc54d081b9ad49af56e99
Ariel Guerra-Adames
69bbc54d081b9ad49af56e9a
Emmanuelle Claeys
69bbc54d081b9ad49af56e9b
Elouan Vuichard
69bbc54d081b9ad49af56e9c
Jean-Michel Loubes
Probing Cultural Signals in Large Language Models through Author Profiling
Large language models (LLMs) are increasingly deployed in applications with societal impact, raising concerns about the cultural biases they encode. We probe these representations by evaluating whether LLMs can perform author profiling from song lyrics in a zero-shot setting, inferring singers' gender and ethnicity without task-specific fine-tuning. Across several open-source models evaluated on more than 10,000 lyrics, we find that LLMs achieve non-trivial profiling performance but demonstrate systematic cultural alignment: most models default toward North American ethnicity, while DeepSeek-1.5B aligns more strongly with Asian ethnicity. This finding emerges from both the models' prediction distributions and an analysis of their generated rationales. To quantify these disparities, we introduce two fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), and show that Ministral-8B displays the strongest ethnicity bias among the evaluated models, whereas Gemma-12B shows the most balanced behavior. Our code is available on GitHub (https://github.com/ValentinLafargue/CulturalProbingLLM).
Large language models exhibit systematic cultural biases when performing author profiling from song lyrics, with varying degrees of ethnic alignment across different models.
large language models zero-shot setting author profiling cultural biases gender inference ethnicity inference fairness metrics Modality Accuracy Divergence Recall Divergence