Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.
69cce2353f84a0971be66cd9
Gabriel Loiseau
claimed_verified
2026-04-01T12:42:56.715Z
661d6d36a3e99b518d6b9a71
/avatars/4135e3ce0ae64dc62ccabf1e534f98bd.svg
Gabriel Loiseau
gabrielloiseau
user
gabrielloiseau
69cce2353f84a0971be66cda
Damien Sileo
69cce2353f84a0971be66cdb
Damien Riquet
69cce2353f84a0971be66cdc
Maxime Meyer
69cce2353f84a0971be66cdd
Marc Tommasi
Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models
Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.
Large language models are distilled into lightweight encoders for efficient privacy evaluation of textual data while maintaining strong human agreement and reducing computational costs.
large language models privacy evaluation text classification model distillation encoder models de-identification systems