Don’t give up 🔥
text
Don’t give up 🔥
new_line
new_line
Do you know what I was planning to do this time last week?
text
Do you know what I was planning to do this time last week?
new_line
new_line
I was preparing to write a report declaring that Jan Nano was a failed project because the benchmark results didn’t meet expectations.
text
I was preparing to write a report declaring that Jan Nano was a failed project because the benchmark results didn’t meet expectations.
new_line
new_line
But I thought — it can’t be. When loading the model into the app, the performance clearly felt better. So why were the benchmark results worse?
text
But I thought — it can’t be. When loading the model into the app, the performance clearly felt better. So why were the benchmark results worse?
new_line
new_line
That’s when I reviewed the entire benchmark codebase and realized something fundamental: agentic or workflow-based approaches introduce a huge gap and variation when benchmarking. Jan-nano was trained with an agentic setup — it simply can’t be benchmarked using a rigid workflow-based method.
text
That’s when I reviewed the entire benchmark codebase and realized something fundamental: agentic or workflow-based approaches introduce a huge gap and variation when benchmarking. Jan-nano was trained with an agentic setup — it simply can’t be benchmarked using a rigid workflow-based method.
new_line
new_line
I made the necessary changes, and the model ended up performing even better than before the issues arose. Turns out the previous benchmarking method conflicted with the way the model was trained.
text
I made the necessary changes, and the model ended up performing even better than before the issues arose. Turns out the previous benchmarking method conflicted with the way the model was trained.
new_line
new_line
What if I had given up? That would’ve meant 1.5 months of training and a huge amount of company resources wasted.
text
What if I had given up? That would’ve meant 1.5 months of training and a huge amount of company resources wasted.
new_line
new_line
But now, this is officially the most successful and biggest release for the whole team — all thanks to Jan-nano.
text
But now, this is officially the most successful and biggest release for the whole team — all thanks to Jan-nano.
new_line
new_line
https://huggingface.co/Menlo/Jan-nano
Menlo/Jan-nano
model
resource
https://huggingface.co/Menlo/Jan-nano
Don’t give up 🔥

Do you know what I was planning to do this time last week?

I was preparing to write a report declaring that Jan Nano was a failed project because the benchmark results didn’t meet expectations.

But I thought — it can’t be. When loading the model into the app, the performance clearly felt better. So why were the benchmark results worse?

That’s when I reviewed the entire benchmark codebase and realized something fundamental: agentic or workflow-based approaches introduce a huge gap and variation when benchmarking. Jan-nano was trained with an agentic setup — it simply can’t be benchmarked using a rigid workflow-based method.

I made the necessary changes, and the model ended up performing even better than before the issues arose. Turns out the previous benchmarking method conflicted with the way the model was trained.

What if I had given up? That would’ve meant 1.5 months of training and a huge amount of company resources wasted.

But now, this is officially the most successful and biggest release for the whole team — all thanks to Jan-nano.

https://huggingface.co/Menlo/Jan-nano
🔥
John6666 Presidentlin Baphomet666 bndp alandao jan-ai