FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
We introduce FIN-bench-v2, a unified benchmark suite for evaluating large language models in Finnish. FIN-bench-v2 consolidates Finnish versions of widely used benchmarks together with an updated and expanded version of the original FIN-bench into a single, consistently formatted collection, covering multiple-choice and generative tasks across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, and alignment. All datasets are converted to HuggingFace Datasets, which include both cloze and multiple-choice prompt formulations with five variants per task, and we incorporate human annotation or review for machine-translated resources such as GoldenSwag and XED. To select robust tasks, we pretrain a set of 2.15B-parameter decoder-only models and use their learning curves to compute monotonicity, signal-to-noise, non-random performance, and model ordering consistency, retaining only tasks that satisfy all criteria. We further evaluate a set of larger instruction-tuned models to characterize performance across tasks and prompt formulations. All datasets, prompts, and evaluation configurations are publicly available via our fork of the Language Model Evaluation Harness at https://github.com/LumiOpen/lm-evaluation-harness. Supplementary resources are released in a separate repository at https://github.com/TurkuNLP/FIN-bench-v2.
6941504965f1e24a1178088d
Joona Kytöniemi
claimed_verified
2025-12-16T14:10:20.947Z
64798c5c4be04c76ce4508db
https://cdn-avatars.huggingface.co/v1/production/uploads/64798c5c4be04c76ce4508db/k-8-2wagNuckcQ4jygu1p.jpeg
Joona Kytöniemi
kjoona
user
kjoona
6941504965f1e24a1178088e
Jousia Piha
claimed_verified
2025-12-25T20:53:12.071Z
643855757de34e8ea7e2745b
/avatars/3a8e7768c48d2822700844e1f6ffdee6.svg
Jousia Piha
poesia
user
poesia
6941504965f1e24a1178088f
Akseli Reunamo
claimed_verified
2025-12-16T22:32:53.707Z
62e77fc7d4b0b8d818b1272b
/avatars/29b62154dc69c8ac66197736561073d0.svg
Akseli Reunamo
akseli-reunamo
user
akseli-reunamo
6941504965f1e24a11780890
Fedor Vitiugin
6941504965f1e24a11780891
Farrokh Mehryary
6941504965f1e24a11780892
Sampo Pyysalo
FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
We introduce FIN-bench-v2, a unified benchmark suite for evaluating large language models in Finnish. FIN-bench-v2 consolidates Finnish versions of widely used benchmarks together with an updated and expanded version of the original FIN-bench into a single, consistently formatted collection, covering multiple-choice and generative tasks across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, and alignment. All datasets are converted to HuggingFace Datasets, which include both cloze and multiple-choice prompt formulations with five variants per task, and we incorporate human annotation or review for machine-translated resources such as GoldenSwag and XED. To select robust tasks, we pretrain a set of 2.15B-parameter decoder-only models and use their learning curves to compute monotonicity, signal-to-noise, non-random performance, and model ordering consistency, retaining only tasks that satisfy all criteria. We further evaluate a set of larger instruction-tuned models to characterize performance across tasks and prompt formulations. All datasets, prompts, and evaluation configurations are publicly available via our fork of the Language Model Evaluation Harness at https://github.com/LumiOpen/lm-evaluation-harness. Supplementary resources are released in a separate repository at https://github.com/TurkuNLP/FIN-bench-v2.
FIN-bench-v2 is a unified benchmark suite for evaluating Finnish large language models, incorporating diverse datasets and evaluation criteria.
large language models benchmark suite reading comprehension commonsense reasoning sentiment analysis world knowledge alignment HuggingFace Datasets cloze multiple-choice pretrain decoder-only models learning curves monotonicity signal-to-noise non-random performance model ordering consistency instruction-tuned models evaluation configurations Language Model Evaluation Harness