Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO
Group Relative Policy Optimization (GRPO) has demonstrated great utilization in post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and, through reinforcement learning, preferred completions are learnt. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and then exchanged in the forms of strings. In this work, we present the first adversarial attack in decentralised GRPO. We demonstrate that malicious parties can poison such systems by injecting arbitrary malicious tokens in benign models in both out-of-context and in-context attacks. Using empirical examples of math and coding tasks, we show that adversarial attacks can easily poison the benign nodes, polluting their local LLM post-training, achieving attack success rates up to 100% in as few as 50 iterations. We propose two ways to defend against these attacks, depending on whether all users train the same model or different models. We show that these defenses can achieve stop rates of up to 100%, making the attack impossible.
69175497a6cea0a2e76c976c
Nikolay Blagoev
claimed_verified
2025-11-17T10:28:17.873Z
66e17afa21dfd93e62ff5433
/avatars/5962116b5198e7ec7c18f4d10982265b.svg
Nikolay Blagoev
Faustify
user
Faustify
69175497a6cea0a2e76c976d
Oğuzhan Ersoy
claimed_verified
2025-11-17T10:28:20.305Z
673c712c8a9ba30a57741a8c
https://cdn-avatars.huggingface.co/v1/production/uploads/673c712c8a9ba30a57741a8c/khBSYmO5V1BM8Q_uPLLWB.jpeg
Oguzhan Ersoy
oguzer
user
oguzer
69175497a6cea0a2e76c976e
Lydia Yiyu Chen
Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO
Group Relative Policy Optimization (GRPO) has demonstrated great utilization in post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and, through reinforcement learning, preferred completions are learnt. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and then exchanged in the forms of strings. In this work, we present the first adversarial attack in decentralised GRPO. We demonstrate that malicious parties can poison such systems by injecting arbitrary malicious tokens in benign models in both out-of-context and in-context attacks. Using empirical examples of math and coding tasks, we show that adversarial attacks can easily poison the benign nodes, polluting their local LLM post-training, achieving attack success rates up to 100% in as few as 50 iterations. We propose two ways to defend against these attacks, depending on whether all users train the same model or different models. We show that these defenses can achieve stop rates of up to 100%, making the attack impossible.
The study identifies and defends against adversarial attacks in decentralized Group Relative Policy Optimization (GRPO) for Large Language Models (LLMs), demonstrating attack success rates of up to 100% and proposing effective defense mechanisms.
Group Relative Policy Optimization GRPO Large Language Models LLMs reinforcement learning decentralised training adversarial attacks malicious tokens out-of-context attacks in-context attacks attack success rates defense mechanisms