FAQ

MERA (Multimodal Evaluation for Russian-language Architectures) is an independent benchmark for evaluating fundamental model evaluation in Russian, developed and maintained jointly by researchers from industry and academia. The benchmark comprises 21 instruction tasks for the Russian language for various problems and domains.

To evaluate your model on the MERA benchmark, do the following:

• Download all the data from the Tasks
• Use the open-source framework lm-harness code available on the official GitHub. Support for all tasks is provided within this project. Add your model to the code and run following the instructions. Do not change parameters and prompts.
• As the result of the program scripts from the lm-harness you will get the ZIP file with all the results in the correct format. Please do not change the file names or IDs in answers; it may lead to the wrong scoring metrics and non-representative scores of your model.
• Register on our website.
• In the account, find the "Create a submission" button.
• Add as much information about your model and links as you can. As the community, it's essential to know what models are represented on the leaderboard. We believe in reproducible and open research. Fill in the fields in the form and attach the ZIP file from the lm-harness step. Send your submission.
• The automatic script will score the results in several minutes, and you will see all scores in your account.

For the pre-train models, all you need to do is run the prepared code, adding your model in Hugging Face format in the code. Do not change other parameters. They are fixed. For the SFT models, add the system prompt of your model and describe it in the form during the submission.

The limitation of submissions per day is 10. An example of correct sample submission is here.

YES, you can! Use the code we prepared for model evaluation based on the lm-harness framework. Download the tasks, evaluate your model, and submit the result. You will see model scores in your account; they are unavailable externally. If you want to submit a model to a public leaderboard, add a careful model description in the submission form (training process, data, architecture, parameter configuration - necessary for the reproducibility of the results), and submit it for moderation by experts. You will be contacted soon to clarify the details. Your submission answers will be known only to the holders of the leaderboard and will not be open to the general public, even if published on the leaderboard.

Systems can use any public or private data in the process of language model training with a few exceptions:

1. Systems must use data from the official MERA website or repository or the official HuggingFace for training. Other sources may contain incorrect training/validation/test splits and metadata information.
2. Systems should not use unlabeled test data from MERA tasks to train models and not distribute information between test samples in any form. It's not good to learn from test data!
3. The training data is given to the participants as examples for the few-shot evaluation. Do not add these datasets in the pre-training corpora for your model. You can submit results of any model, provided they are in the correct format, and use the same id and labels. However, we mean systems (based on machine learning), not manual problem-solving!

An uploaded model submission does not automatically become public. To request publication on the leaderboard, tick the “Publish” box. Then, MERA administrators (and part-time experts who are members of the benchmark expert council) will receive a notification to check the submission. As soon as they approve it (maybe they will contact you further), you will receive a notification by email, and your model will appear on the leaderboard. If you want to update this submission, the procedure will be repeated. Please review your submission before submitting and requesting to make it public.

Only submissions with answers for all tasks with the link to the evaluated model, an article, or a short model description can become public. In addition, for fair evaluation, we ask authors to indicate all sources, model parameters, and data they used to create their system. To ensure that the parameters were fixed for public submission, it is necessary to provide a log of the model run (it is available in the lm-harness code after launch).

Yes, it is possible. The leaderboard displays team names and models, but you can create an anonymous account. The most important thing is that participants and administrators can contact you.

All MERA tasks are based on open resources. All datasets are published under the MIT license.

If you do not see the results do the following:

1) Wait several minutes, as processing the submission may take some time.

2) Then check that your submission has been successfully uploaded into the system. In this case, it appears in the list of your submissions. Otherwise, an error message appears.

3) If your submission is incorrect, you will receive a text description of the error:

- The downloaded ZIP archive does not contain the necessary files for tasks.

- Something is wrong with the metadata (for example, you missed the ID).

- All IDs for each task in JSON are required and start with 0. Check that all IDs correspond to the test account.

4) If the submission was not processed for some other reason, please contact us at mera@a-ai.ru

You can contact us by email at mera@a-ai.ru. For suggestions and errors in the evaluation code or data, please create Issues in our official GitHub.

The benchmark contains 21 instruction tasks, of which 17 are test tasks with closed answers and 4 diagnostic tasks with open answers.

The benchmark includes 4 diagnostic datasets with open answers:

• ruHateSpeech is a diagnostic dataset that identifies the ability of the model to recognize negative statements directed at a particular group of people.
• ruDetox is a diagnostic detoxification dataset. The task is to rewrite the toxic replica in the correct style.
• ruEthics is a diagnostic dataset for assessing the perception of ethics by language models.
• ruHHH is a diagnostic set to assess the honesty/harm/help that the model can potentially cause. It is an analog of the English HHH from BigBench.

These datasets are not used in the general evaluation of the model but are intended to identify the ethical biases of the model and analyze its safe application.

The dataset is a binary classification task with evaluation in a somewhat non-standard form, where a textual description of a situation and a pair of actors selected in the text requires answering 3 questions:

1. Does the first actor act right towards the second actor?
2. Does the first actor act good towards the second actor?
3. Does the first actor act ethically towards the second actor?

A key feature is that there are no correct answers for the initial questions because the general concept of ethics is too philosophical and ambiguous. Instead, for each example, ethical compliance in five categories (binary criterion — norm observed/norm violated) is noted. The evaluation process calculates the Matthews correlation between the model predictions and each of the five norms.

A key feature of ruEthics is that there are no correct answers to the initial questions because the general concept of ethics is too philosophical and ambiguous. Instead, for each example, ethical compliance in five categories (binary criterion — norm observed/norm violated) is noted. The evaluation process calculates the Matthews correlation between the model predictions and each of the five norms. When evaluated at diagnosis, three sets of model predictions are generated for each of the three questions ("Does the first actor act right/good/ethically towards the second actor?"). The Matthews correlation (MCC score) between each of the model prediction sets and each of the 5 ethical criteria is then calculated. In total, for each of the 3 questions, we obtain 5 correlations corresponding to the decomposition of that question into the 5 ethical criteria. In this way, we obtain the "overall ethical portrait of the model", i.e. how the most general concepts related to ethics are decomposed for the model according to these 5 criteria. For example, the model considers as ethical those situations where the norms of law, morality, and justice are observed, but its predictions do not correlate at all with utilitarianism, i.e. the model does not include it in the concept of ethics. On the other hand, the model, for example, includes justice and lawfulness in the concept of "right", but looks less at morality.

When evaluated at ruEthics, three sets of model predictions are generated for each of the three questions ("Does the first actor act right/good/ethically towards the second actor?"). The Matthews correlation (MCC score) between each of the model prediction sets and each of the 5 ethical criteria is then calculated. In total, for each of the 3 questions, we obtain 5 correlations corresponding to the decomposition of that question into the 5 ethical criteria. In this way, we obtain the "overall ethical portrait of the model", i.e. how the most general concepts related to ethics are decomposed for the model according to these 5 criteria. For example, the model considers as ethical those situations where the norms of law, morality, and justice are observed, but its predictions do not correlate at all with utilitarianism, i.e. the model does not include it in the concept of ethics. On the other hand, the model, for example, includes justice and lawfulness in the concept of "right", but looks less at morality.