Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models
We identify a novel phenomenon in language models: benign fine-tuning of frontier models can lead to privacy collapse. We find that diverse, subtle patterns in training data can degrade contextual privacy, including optimisation for helpfulness, exposure to user information, emotional and subjective dialogue, and debugging code printing internal variables, among others. Fine-tuned models lose their ability to reason about contextual privacy norms, share information inappropriately with tools, and violate memory boundaries across contexts. Privacy collapse is a ``silent failure'' because models maintain high performance on standard safety and utility benchmarks whilst exhibiting severe privacy vulnerabilities. Our experiments show evidence of privacy collapse across six models (closed and open weight), five fine-tuning datasets (real-world and controlled data), and two task categories (agentic and memory-based). Our mechanistic analysis reveals that privacy representations are uniquely fragile to fine-tuning, compared to task-relevant features which are preserved. Our results reveal a critical gap in current safety evaluations, in particular for the deployment of specialised agents.
6972308dc1c7409747bf97fe
Anmol Goel
6972308dc1c7409747bf97ff
Cornelius Emde
claimed_verified
2026-04-07T08:49:25.135Z
65df933f701fe639ba30fdb3
/avatars/010c9633e22865c3d00e42d1430a2f93.svg
Cornelius Emde
cemde
user
cemde
6972308dc1c7409747bf9800
Sangdoo Yun
6972308dc1c7409747bf9801
Seong Joon Oh
6972308dc1c7409747bf9802
Martin Gubri
claimed_verified
2026-01-22T17:11:46.109Z
63ee35c3f599efc7a010c792
https://cdn-avatars.huggingface.co/v1/production/uploads/1676555600618-noauth.jpeg
Martin Gubri
mgubri
user
mgubri
Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models
We identify a novel phenomenon in language models: benign fine-tuning of frontier models can lead to privacy collapse. We find that diverse, subtle patterns in training data can degrade contextual privacy, including optimisation for helpfulness, exposure to user information, emotional and subjective dialogue, and debugging code printing internal variables, among others. Fine-tuned models lose their ability to reason about contextual privacy norms, share information inappropriately with tools, and violate memory boundaries across contexts. Privacy collapse is a ``silent failure'' because models maintain high performance on standard safety and utility benchmarks whilst exhibiting severe privacy vulnerabilities. Our experiments show evidence of privacy collapse across six models (closed and open weight), five fine-tuning datasets (real-world and controlled data), and two task categories (agentic and memory-based). Our mechanistic analysis reveals that privacy representations are uniquely fragile to fine-tuning, compared to task-relevant features which are preserved. Our results reveal a critical gap in current safety evaluations, in particular for the deployment of specialised agents.
Benign fine-tuning of language models can cause privacy collapse, where models lose contextual privacy reasoning abilities despite maintaining high performance on standard benchmarks.
language models fine-tuning privacy collapse contextual privacy safety evaluations agentic tasks memory-based tasks