As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from 
text
As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from 
@Rijgersberg
mention
Rijgersberg
 et al!)
text
 et al!)
new_line
new_line
Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 
text
Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 
new_line
new_line
Does LLM, as a judge, work outside of English?
text
Does LLM, as a judge, work outside of English?
new_line
new_line
Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 
text
Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 
new_line
new_line
What we want to know is:
text
What we want to know is:
new_line
- does auto/LLM eval work in general for a particular language
text
- does auto/LLM eval work in general for a particular language
new_line
- which model(s) works best as a judge
text
- which model(s) works best as a judge
new_line
- do LLMs' judgments of non-English models match human preferences? 
text
- do LLMs' judgments of non-English models match human preferences? 
new_line
new_line
We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: 
text
We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: 
https://github.com/huggingface/data-is-better-together/issues/61
https://github.com/huggingface/data-is-better-together/issues/61
link
text
new_line
new_line
Other ideas...
text
Other ideas...
new_line
new_line
Could an approach like 
text
Could an approach like 
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models (2404.18796)
https://huggingface.co/papers/2404.18796
2404.18796
paper
resource
https://huggingface.co/papers/2404.18796
 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
text
 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
As part of the Data is Better Together MPEP project, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts into a new target language (amazing work from @Rijgersberg et al!)

Our next step is to use these translated prompts to evaluate the performance of LLMs for non English languages. 

Does LLM, as a judge, work outside of English?

Ideally, it would be compelling to leverage LLMs to judge models for non-English since this significantly lowers the barrier to evaluating models (although it doesn't remove this barrier altogether). 

What we want to know is:
- does auto/LLM eval work in general for a particular language
- which model(s) works best as a judge
- do LLMs' judgments of non-English models match human preferences? 

We're starting to think about how to approach this. If you have any ideas of possible approaches feel free to comment or join the discussion here: https://github.com/huggingface/data-is-better-together/issues/61 

Other ideas...

Could an approach like https://huggingface.co/papers/2404.18796 with the SOA models for a particular language work? i.e., choose 4 of the best open LLMs for Arabic and use those at the pool of raters rather than relying on one powerful judge LLM? 
🤗
NePe kristaller486 mmhamdy Rijgersberg alvarobartt
🔥
NePe mmhamdy