This paper (
text
This paper (
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs (2412.18925)
https://huggingface.co/papers/2412.18925
2412.18925
paper
resource
https://huggingface.co/papers/2412.18925
) has a really interesting recipe for inducing o1-like behaviour in Llama models:
text
) has a really interesting recipe for inducing o1-like behaviour in Llama models:
new_line
new_line
* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting.
text
* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting.
new_line
* Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases)
text
* Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases)
new_line
* Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1
text
* Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1
new_line
* Use the resulting data for SFT & RL
text
* Use the resulting data for SFT & RL
new_line
* Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.
text
* Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.
new_line
new_line
Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
text
Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
This paper (https://huggingface.co/papers/2412.18925) has a really interesting recipe for inducing o1-like behaviour in Llama models:

* Iteratively sample CoTs from the model, using a mix of different search strategies. This gives you something like Stream of Search via prompting.
* Verify correctness of each CoT using GPT-4o (needed because exact match doesn't work well in medicine where there are lots of aliases)
* Use GPT-4o to reformat the concatenated CoTs into a single stream that includes smooth transitions like "hmm, wait" etc that one sees in o1
* Use the resulting data for SFT & RL
* Use sparse rewards from GPT-4o to guide RL training. They find RL gives an average ~3 point boost across medical benchmarks and SFT on this data already gives a strong improvement.

Applying this strategy to other domains could be quite promising, provided the training data can be formulated with verifiable problems!
🔥
lvwerra plaguss John6666 davidquicast CoffeeBliss bird-of-paradise
66bcff7ddac7d713269de681
https://cdn-avatars.huggingface.co/v1/production/uploads/66bcff7ddac7d713269de681/AwDil0zXAi29lTftRNB3u.png
CoffeeBliss
CoffeeBliss
user