How MERA works
Modern large language models (like ChatGPT, Llama, YandexGPT, and GigaChat) are developing rapidly and require objective comparison and independent evaluation.
We observe from international experience that model evaluation is conducted on different benchmarks and in various experimental setups and scenarios, leading to a lack of understanding of what models can genuinely accomplish and making it impossible to compare the models' abilities. Openness and transparency in the evaluation process are the key issues because any model will be assessed by the developer according to their methodology claiming the superiority of their models.
We have developed an open instructional benchmark for evaluating large language models for the Russian language. The benchmark website provides a ranking of models based on their performance in solving a fixed set of tasks compiled by experts with standardized configurations of prompts and parameters.
We propose a new methodology to evaluate SOTA language models:
It includes a wide range of challenging tests focused on critical professional areas such as agriculture and medicine. All tasks in this set were created by leading experts in the fields of agriculture and medicine, edited by professional editors, then manually retested one by one by three experts.
How are task prompts designed?
For each task, experts have manually created several diverse universal model-agnostic instruction prompts with clearly defined requirements for the answer output format. These prompts are uniformly distributed among all questions in the task so that each question is assigned exactly one prompt.
This format allows to obtaining an average score based on different prompts and all models are evaluated in equal conditions: prompts do not "favor" specific models. For these reasons, instructions as well as generation parameters and few-shot examples cannot be changed within model evaluation.
How is measurement done?
The benchmark's scoring system is based on the international LM Evaluation Harness framework, which provides generative and log-likelihood setup for model evaluation.
| Generative evaluation | Log-likelihood evaluation |
|---|---|
| Does not require access to logits, suitable for any model capable of generating text. | Is not suitable for API models since they typically do not return logits used in log-likelihood evaluation. |
| Requires answer post-processing (no universal heuristics, human SBS and LLM-as-a-Judge / special parsers). | Does not require model response post-processing because the answer is a fixed letter or number. |
| Smaller models tend to generate irrelevant responses. | Allows to measure the probability of specific responses from a language model. |
| We recommend to evaluate instructive models (SFT-like) and APIs only in generative setup. | Better suited for evaluation of pretrained models and smaller models. |
Helpful links
Repository:
Research paper:
HF datasets:
Technical support:
FAQ
HF datasets 2.0:
FAQ
What is MERA Multi?
MERA Multi is a multimodal task benchmark within the independent MERA benchmark. The 18 instructional tasks include image and video analysis, speech and non-speech sound recognition. These skills are tested in free-form short-answer and multiple-choice tasks
How do I add my result to the public leaderboard?
By default, uploaded submissions remain private. To make the result public, check the "Publish" option. After that, MERA Code administrators will receive a request for review. If the submission meets the requirements, it will be approved and you will receive an email notification. Your model will appear on the leaderboard. If necessary, you may be contacted for clarification of details.To be accepted for publication, a submission must contain:
— Results for all tasks;
— Description of the solution (at least a link to the model, article, or model description);
— A complete list of resources used (data sources, model parameters, and other key details).
If you update your submission, the review process will be repeated. Before sending, make sure all the details are correct and the description is as detailed as possible
How do I add my result to the public leaderboard?
To evaluate your model on the MERA benchmark, collect the results of its runs for each dataset.
— Use the official evaluation code base from the official project repository. Add your model to the code and run the test according to the instructions. Do not change the startup parameters!
— As a result of the code, you will receive a submission in the form of a ZIP archive for uploading to the site. Do not change the name of the files or ID in the responses in the submissions, otherwise the evaluation result will be incorrect.
— Register on the website. Create a new submission in your account. Add as much information about your model as possible, and provide links to your solution (article or code on github). This is important! In order to get on the leaderboard, we need to make sure that your result is fair. We believe that science should be reproducible!
— Attach a ZIP archive. Send the submission to the system. In a few hours, the automatic script will process the data, and the result will be available in your personal account
Can I test my proprietary model on MERA?
Yes, you can! We have prepared code for evaluation via the lm-harness framework, including for API models. Run testing of your model and upload the archive with the results to the site. The scores will be available to you in your personal account and will remain closed to other users. If you want to place your model on a public leaderboard, please provide as much information about it as possible when submitting:
— training process,— data used,
— architecture,
— system parameters.
This information will help the community understand and reproduce your system. The submission is moderated by experts who can contact you for further details.
Important: even if your model is published in the ranking, its answers will remain available only to experts and will not be disclosed to the general public
Is it possible to make an anonymous submission on a public leaderboard?
Yes, you can. The leaderboard displays the names of teams and models, but you can make an anonymous account. The main thing is that participants and administrators can contact you
What license are the datasets available under?
Datasets created using open-source data inherit the original license (mostly CC-BY-4.0). Images, audio, and video collected specifically for MERA Multi are protected by a special MERA license: the materials may only be used for model testing, not training. Such images, audio, and video are watermarked, preventing their "accidental" use in model training.The materials are permitted for educational use, but only for model evaluation
Why can't I see the results of my model/submission?
If you've submitted a submission for evaluation, please wait first — model processing may take some time.Then check that your submission has loaded in the system — it will appear in your list of submissions. Otherwise, an error message will appear.In other cases, if your submission fails for some reason, please contact us at mera@a-ai.ruIf the submission is invalid, the system will display a text description of the error, which covers cases such as:— The uploaded ZIP archive is missing one of the required files for the tasks.— There's something wrong with the metadata (for example, you're missing an ID). All IDs for each task in the JSON are required and start with 0. Check that all IDs match the test set
I found a bug, I have suggestions and comments!
You can contact us by email at mera@a-ai.ru. For suggestions and errors in the evaluation code or data, please create Issues in our official GitHub.For quick communication, there is a Telegram chat for technical support
How are the results aggregated on the leaderboard?
We have introduced three key metrics that reflect model quality:
— Attempted Score — quality on solved tasks: modality weights are recalculated only across those tasks the model actually attempted.
— Coverage — the proportion of processed tasks within the leaderboard.
— Total Score — the final score that determines the ranking, calculated as Total Score = Attempted Score × Coverage.
The Total Score is used to rank models on the leaderboards. This approach already enables the creation of a multimodal model ranking that accounts for different task types. In the future, it will allow for seamless benchmark expansion by adding new and more complex scenarios without compromising compatibility with existing results.
To which leaderboard should I submit my model?
For the multimodal version of MERA, we introduced four independent leaderboards:
1. Images — tasks focused on analyzing photos, illustrations, and screenshots.
2. Audio — tasks involving speech, music, and ambient sound understanding.
3. Video — tasks for analyzing video content without audio.
4. Multi — a unified leaderboard combining tasks from all modalities.
Participants can choose which leaderboard to submit their model to.
To qualify, a submission only needs to include at least one task from the selected leaderboard.
Partial submissions are allowed to ensure that evaluation remains fair and transparent.
mera@a-ai.ru
GitHub
hugging face