How Robust Is Your Model in Complex Code Generation Tasks? 🤔
text
How Robust Is Your Model in Complex Code Generation Tasks? 🤔
new_line
new_line
We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.
text
We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.
new_line
new_line
It seems that the Claude 3 models outperforme ChatGPT:
text
It seems that the Claude 3 models outperforme ChatGPT:
new_line
Model                                             / Avg. (pass@3)
text
Model                                             / Avg. (pass@3)
new_line
Claude 3 Haiku                          / 27.67
text
Claude 3 Haiku                          / 27.67
new_line
GPT-3.5-Turbo                            / 23.75
text
GPT-3.5-Turbo                            / 23.75
new_line
Mixtral-8x22B-Instruct-v0.1  / 8.35
text
Mixtral-8x22B-Instruct-v0.1  / 8.35
new_line
new_line
Read our Preprint📃: 
text
Read our Preprint📃: 
PECC: Problem Extraction and Coding Challenges (2404.18766)
https://huggingface.co/papers/2404.18766
2404.18766
paper
resource
https://huggingface.co/papers/2404.18766
new_line
Look at the dataset🔎: 
text
Look at the dataset🔎: 
https://huggingface.co/datasets/PatrickHaller/pecc
PatrickHaller/pecc
dataset
resource
https://huggingface.co/datasets/PatrickHaller/pecc
new_line
new_line
We also got accepted at LREC-COLING '24  🎉
text
We also got accepted at LREC-COLING '24  🎉
How Robust Is Your Model in Complex Code Generation Tasks? 🤔

We've launched the PECC benchmark to challenge chat models in code generation, drawing from the Advent of Code for programming tasks and the Euler Project for math-heavy challenges. This new task tests models with problems presented in both detailed prose and concise "leet code" styles, evaluating their ability to understand and solve complex coding issues and math problem in chat-based interactions.

It seems that the Claude 3 models outperforme ChatGPT:
Model                                             / Avg. (pass@3)
Claude 3 Haiku                          / 27.67
GPT-3.5-Turbo                            / 23.75
Mixtral-8x22B-Instruct-v0.1  / 8.35

Read our Preprint📃: https://huggingface.co/papers/2404.18766
Look at the dataset🔎: https://huggingface.co/datasets/PatrickHaller/pecc

We also got accepted at LREC-COLING '24  🎉
🔥
victor barthfab alanakbik