How MERA works
Modern large language models (such as ChatGPT, Llama, YandexGPT, GigaChat) are actively developing and need fair comparison and independent evaluation. There is no single standard for evaluation, and therefore the models cannot be compared fairly, as the measurements are performed in disparate experimental settings (different data for evaluation, methods of measurement). Openness and transparency of the procedure is a key problem of evaluation, also because models are usually evaluated by developers who are interested in their models getting high scores. We present a Russian-language industrial benchmark for comprehensive validation of large language models in the agriculture and medical and healthcare industries. The benchmark website ranks models on the quality of solving a fixed set of tasks composed by experts, with standardized configurations of prompts and parameters. The project is supported by the AI Alliance, leading industry players and academic partners who research language models.
We propose a new methodology to evaluate SOTA language models:
It includes a wide range of challenging tests focused on critical professional areas such as agriculture and medicine. All tasks in this set were created by leading experts in the fields of agriculture and medicine, edited by professional editors, then manually retested one by one by three experts.
How are task prompts designed?
For each task, experts have manually created several diverse universal model-agnostic instruction prompts with clearly defined requirements for the answer output format. These prompts are uniformly distributed among all questions in the task so that each question is assigned exactly one prompt.
This format allows to obtaining an average score based on different prompts and all models are evaluated in equal conditions: prompts do not "favor" specific models. For these reasons, instructions as well as generation parameters and few-shot examples cannot be changed within model evaluation.
Helpful links
Research paper:
Technical support:
FAQ:
Repository:
HF datasets:
GitVerse:
FAQ
What is MERA?
MERA Industrial is an independent Russian-language benchmark for testing large language models in real-world industry scenarios. The pilot directions for the project were chosen as medicine and agriculture. The benchmark, created by leading industry experts, includes a wide range of complex specialized tasks and will be scaled to new areas
How to use the MERA benchmark?
To evaluate your model on the MERA Industrial benchmark, gather its run results for every dataset.
Use the evaluation code provided in the official benchmark GitHub. Support for all tasks is provided within this project. Add your model to the code and run following the instructions. Do not change parameters and prompts.
As the result of the program scripts from the lm-harness you will get the ZIP file with all the results in the correct format. Please do not change the file names or IDs in answers; it may lead to the wrong scoring metrics and non-representative scores of your model.
Register on our website.
In the account, find the "Create a submission" button.
Add as much information about your model and links as you can. As the community, it's essential to know what models are represented on the leaderboard. We believe in reproducible and open research. Fill in the fields in the form and attach the ZIP file from the lm-harness step. Send your submission.
The automatic script will score the results in several minutes, and you will see all scores in your account.
Can I evaluate a private model on the MERA benchmark?
YES, you can! Use the code we prepared for model evaluation based on the lm-harness framework. Download the tasks, evaluate your model, and submit the result. You will see model scores in your account; they are unavailable externally. If you want to submit a model to a public leaderboard, add a careful model description in the submission form (training process, data, architecture, parameter configuration - necessary for the reproducibility of the results), and submit it for moderation by experts. You will be contacted soon to clarify the details. Your submission answers will be known only to the holders of the leaderboard and will not be open to the general public, even if published on the leaderboard.
Is it possible to evaluate models using API?
Yes! The only requirement is to add support for your model to the codebase. You can find the official lm-evaluation-harness guide on integrating a new API into the framework here https://github.com/MERA-Evaluation/MERA_Industrial
How can I add my model result on public leaderboard?
An uploaded model submission does not automatically become public. To request publication on the leaderboard, tick the “Publish” box.
You can choose a target leaderboard (e.g., "Agriculture," "Medicine") or submit to all available boards. After this, the MERA administrators (who are also members of the benchmark's expert council) will receive a verification request.
Publication Requirements:
For a submission to be approved and for your model to appear on the leaderboard, it must contain:
Results for all core tasks of the selected leaderboard. If you are submitting to a public leaderboard, all tasks within it must be tested—submissions with missing results are not permitted.
A solution description (at a minimum, a link to the model, a research paper, or a detailed description of the model and its training process).
A complete list of utilized resources (data sources, model parameters, and other key details).
If the submission meets all requirements, it will be approved, and you will receive a notification via email. If necessary, we may contact you to clarify details.
Important: If you update your submission, the verification process will restart. Before submitting, please ensure all data is correct and the description is as detailed as possible.
Are there any limitations for model submissions?
Systems can use any public or private data in the process of language model training with a few exceptions:
Systems must use data from the official MERA website or repository or the official HuggingFace for training. Other sources may contain incorrect training/validation/test splits and metadata information.
Systems should not use unlabeled test data from MERA tasks to train models and not distribute information between test samples in any form. It's not good to learn from test data!
The training data is given to the participants as examples for the few-shot evaluation. You can submit results of any model, provided they are in the correct format, and use the same id and labels. However, we mean systems (based on machine learning), not manual problem-solving!
Is it possible to make an anonymous submission on the public leaderboard?
Yes, it is possible. The leaderboard displays team names and models, but you can create an anonymous account. The most important thing is that participants and administrators can contact you.
What license do your datasets have?
All MERA Industrial tasks are based on open resources. All datasets are published under the MIT license.
Why do I not see my submission/model results?
If you do not see the results do the following:
1) Wait several minutes, as processing the submission may take some time.
2) Then check that your submission has been successfully uploaded into the system. In this case, it appears in the list of your submissions. Otherwise, an error message appears.
3) If your submission is incorrect, you will receive a text description of the error:
- The downloaded ZIP archive does not contain the necessary files for tasks.
- Something is wrong with the metadata (for example, you missed the ID).
- All IDs for each task in JSON are required and start with 0. Check that all IDs correspond to the test account.
4) If the submission was not processed for some other reason, please contact us at mera@a-ai.ru
I found a bug. I have suggestions and comments!
You can contact us by email at mera@a-ai.ru. For suggestions and errors in the evaluation code or data, please create Issues in our official GitHub https://github.com/MERA-Evaluation/MERA_Industrial
How many jobs are there in MERA Industrial?
MERA Industria has 3 tasks — one medical and two related to agriculture. Each task belongs to one domain within its industry. Inside each task, there is a division into topics related to that domain. All tasks consist of closed answers.
mera@a-ai.ru
hugging face