Multi-metric evaluation tool for Turkish LLM v0.2 — cost + hallucination
#14
by vigneshwar234 - opened
Merhaba! Hi Turkish LLM community!
Great to see v0.2 of the leaderboard! For teams choosing Turkish LLMs for production, I built a complementary framework that adds cost and quality dimensions.
LLM Evaluation Framework:
- Cost per 1K tokens — Turkish vs frontier model cost comparison
- Hallucination Rate — critical for Turkish news, legal, enterprise use cases
- Latency p95 — API latency from Turkey/EU
- Accuracy — MMLU + TruthfulQA + custom Turkish-language benchmark support
- Reasoning Quality — chain-of-thought depth
Works with any LiteLLM-compatible model.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source. Happy to discuss adding Turkish-specific evaluation tasks!