Multimodal Evaluation of Russian-language Architectures
Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (image-to-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking and licenses for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.
6920b3b2b5612535ed9554f9
Artem Chervyakov
6920b3b2b5612535ed9554fa
Ulyana Isaeva
claimed_verified
2025-11-27T20:40:04.347Z
6207bbdb2f97dabc41b3b32b
/avatars/6e568045b0ebaa05ac1d469d941d4c96.svg
Ulyana
ulyanaisaeva
user
ulyanaisaeva
6920b3b2b5612535ed9554fb
Anton Emelyanov
6920b3b2b5612535ed9554fc
Artem Safin
6920b3b2b5612535ed9554fd
Maria Tikhonova
6920b3b2b5612535ed9554fe
Alexander Kharitonov
claimed_verified
2025-11-28T13:49:07.209Z
65265ffb193ab5e06b5c3bc1
/avatars/d02fd8bd4ea27be2c4f7699697d922bd.svg
Alexander Kharitonov
mathamateur
user
mathamateur
6920b3b2b5612535ed9554ff
Yulia Lyakh
6920b3b2b5612535ed955500
Petr Surovtsev
6920b3b2b5612535ed955501
Denis Shevelev
6920b3b2b5612535ed955502
Vildan Saburov
6920b3b2b5612535ed955503
Vasily Konovalov
claimed_verified
2025-11-27T20:40:06.412Z
622b1f6b9f6139daa8e998ce
/avatars/842719c100a5969be75d04da97333675.svg
Vasily Konovalov
Vasily
user
Vasily
6920b3b2b5612535ed955504
Elisei Rykov
claimed_verified
2025-11-27T20:40:09.480Z
636b867ecde3707d10999b96
https://cdn-avatars.huggingface.co/v1/production/uploads/1667991162387-noauth.png
Elisei Rykov
lmeribal
user
lmeribal
6920b3b2b5612535ed955505
Ivan Sviridov
claimed_verified
2025-11-27T10:00:36.928Z
61dedb1b2066746d68b63adb
https://cdn-avatars.huggingface.co/v1/production/uploads/61dedb1b2066746d68b63adb/PLBHQxvbcay3qDJjY7HM3.jpeg
Ivan Sviridov
univanxx
user
univanxx
6920b3b2b5612535ed955506
Amina Miftakhova
6920b3b2b5612535ed955507
Ilseyar Alimova
6920b3b2b5612535ed955508
Alexander Panchenko
6920b3b2b5612535ed955509
Alexander Kapitanov
6920b3b2b5612535ed95550a
Alena Fenogenova
Multimodal Evaluation of Russian-language Architectures
Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (image-to-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking and licenses for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.
Mera Multi is an open multimodal evaluation framework for Russian-spoken architectures, addressing the lack of such benchmarks with 18 newly constructed tasks and a methodology to prevent benchmark leakage.
multimodal large language models Mera Multi multimodal evaluation framework Russian language multimodal abilities benchmark leakage watermarking