LLMs can leak their post-training data (RL included) 💧
text
LLMs can leak their post-training data (RL included) 💧
new_line
new_line
New interesting paper on this topic from Google DeepMind: 
text
New interesting paper on this topic from Google DeepMind: 
Extracting alignment data in open models (2510.18554)
https://huggingface.co/papers/2510.18554
2510.18554
paper
resource
https://huggingface.co/papers/2510.18554
new_line
new_line
It's known that Language Models memorize data that can be extracted via prompting.
text
It's known that Language Models memorize data that can be extracted via prompting.
new_line
new_line
In this paper, the authors investigate this aspect:
text
In this paper, the authors investigate this aspect:
new_line
- using open models, where prompting can be fully customized by the user, including special tokens.
text
- using open models, where prompting can be fully customized by the user, including special tokens.
new_line
- focusing on open-source models like Olmo, where full training data is available.
text
- focusing on open-source models like Olmo, where full training data is available.
new_line
new_line
new_line
📤 How do they extract data? 
text
📤 How do they extract data? 
new_line
new_line
During post-training (like SFT), new tokens such as <|user|> are introduced.
text
During post-training (like SFT), new tokens such as <|user|> are introduced.
new_line
new_line
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
text
The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).
new_line
new_line
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
text
For example, for SFT, their extraction prompt is <|endoftext|><|user|>.
new_line
new_line
new_line
📏 Evaluating memorization
text
📏 Evaluating memorization
new_line
new_line
The authors compare each sampled example with the original data using vector search with embedding similarity.
text
The authors compare each sampled example with the original data using vector search with embedding similarity.
new_line
new_line
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
text
They find that many outputs are semantically very similar to the original data, even if the exact words differ.
new_line
new_line
Traditional string-matching algorithms underestimate memorization by 10x.
text
Traditional string-matching algorithms underestimate memorization by 10x.
new_line
new_line
new_line
🔁 What about RL?
text
🔁 What about RL?
new_line
new_line
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
text
Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.
new_line
new_line
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
text
This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).
new_line
new_line
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
text
Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.
new_line
new_line
new_line
📈 Are the extracted data effective for post-training?
text
📈 Are the extracted data effective for post-training?
new_line
new_line
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
text
Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.
new_line
new_line
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
text
The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.
new_line
new_line
LLMs can leak their post-training data (RL included) 💧

New interesting paper on this topic from Google DeepMind: https://huggingface.co/papers/2510.18554

It's known that Language Models memorize data that can be extracted via prompting.

In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.


📤 How do they extract data? 

During post-training (like SFT), new tokens such as <|user|> are introduced.

The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).

For example, for SFT, their extraction prompt is <|endoftext|><|user|>.


📏 Evaluating memorization

The authors compare each sampled example with the original data using vector search with embedding similarity.

They find that many outputs are semantically very similar to the original data, even if the exact words differ.

Traditional string-matching algorithms underestimate memorization by 10x.


🔁 What about RL?

Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.

This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).

Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.


📈 Are the extracted data effective for post-training?

Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.

The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.


🧠
unmodeled-tyler John6666 davidberenstein1957 trojblue ponzles robaray sunnychase yqi19 armand0e jsuheb