Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction
The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.
69a550ffbc2d66e68888f620
Nils Schwager
claimed_verified
2026-03-02T09:36:58.823Z
6446bb654705822626e29305
https://cdn-avatars.huggingface.co/v1/production/uploads/6446bb654705822626e29305/enEiepYovZS4S3A6jVOb8.jpeg
Nils Schwager
nsschw
user
nsschw
69a550ffbc2d66e68888f621
Simon Münker
claimed_verified
2026-03-06T13:57:23.236Z
64be8523e90972519da222e4
https://cdn-avatars.huggingface.co/v1/production/uploads/64be8523e90972519da222e4/h2GM63roX58oBy3YqKuZK.jpeg
Simon Münker
simon-muenker
user
simon-muenker
69a550ffbc2d66e68888f622
Alistair Plum
69a550ffbc2d66e68888f623
Achim Rettinger
Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction
The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.
Conditioned Comment Prediction evaluates LLMs' ability to simulate social media user behavior, revealing that supervised fine-tuning affects textual structure but not semantic accuracy, and that explicit conditioning becomes unnecessary when models can infer from behavioral histories.
Conditioned Comment Prediction Large Language Models supervised fine-tuning prompting strategies behavioral histories semantic grounding textual structure