The AI Alliance has launched an open leaderboard to evaluate models’ reasoning capabilities in Russian.

The AI Alliance has launched the public MERA TEXT leaderboard to evaluate the ability of modern language models to reason in Russian. The new leaderboard brings together four sets of mathematical and logical problems covering different types of reasoning and enables models to be compared in a unified, reproducible environment.

Over the past year, the focus of large language model development has shifted significantly. While models previously competed primarily in the breadth of their knowledge, their ability to answer questions, and their coding skills, one of today’s key areas of progress is the ability to construct long chains of reasoning, identify non-obvious solutions, and handle complex problem settings.

This creates a new challenge: how can these capabilities be measured objectively?

Why Mathematical Problems Are Used for Evaluation

Reasoning is not limited to mathematics. However, mathematical problems make it possible to assess objectively whether a model can arrive at the correct answer through a sequence of logical steps rather than simply reproduce information encountered during training.

A high-quality reasoning test should:

• require multiple sequential steps;
• have an unambiguously verifiable answer;
• support reproducible model comparison;
• include problems of different types and difficulty levels.

A wide range of specialized mathematical datasets and leaderboards already exists for English. For Russian, however, there are still far fewer open platforms that allow modern models to be compared under identical conditions.

The new MERA leaderboard is designed to address this gap.

Four Problem Sets for Different Types of Reasoning

A model’s reasoning capabilities cannot be evaluated using a single dataset. Some problems require a non-standard mathematical insight, while others involve long calculations, tracking dozens of intermediate states, or correctly processing large volumes of interconnected information.

For this reason, the leaderboard includes four problem sets: Luzitania, TMath, MMReD, and ruAIME.

Luzitania

Luzitania is a collection of 251 advanced mathematical problems developed using the rules of the AIMO3 competition. It includes problems from leading national and international mathematical olympiads, materials from the Tournament of Towns, and advanced abstract mathematics problems from the MathArena and MathArxiv datasets.

When building the dataset, preference was given to problems originally published in languages other than English and therefore less likely to appear in the training data of modern models. All problems were translated into Russian and partially reformulated.

The dataset includes only problems that modern open models solve inconsistently and that require long chains of reasoning. Luzitania evaluates a model’s ability to select an appropriate solution strategy, maintain a large number of intermediate steps, and use external tools only when they are genuinely helpful.

The dataset was created by the Lomonosov Research Institute.

TMath

TMath brings together 310 problems from the Russian National Mathematics Olympiad and the Moscow Mathematical Olympiad from 1998 to 2025.

The dataset covers algebra, geometry, combinatorics, number theory, and other areas of mathematics. Particular emphasis is placed on problems that cannot be solved through brute-force search or the direct application of a known formula. Instead, a model must independently identify the key idea and consistently develop it into a complete solution.

Problems requiring images, allowing multiple correct answers, or involving open-ended responses were excluded. Additional filtering removed overly simple problems, preserving the benchmark’s ability to distinguish between strong models.

The dataset was created by T-Bank.

MMReD

MMReD is a synthetic benchmark for reasoning in dense-context settings. Unlike traditional tests that require locating a single fact within a long text, every part of the input sequence is relevant in MMReD.

The model receives a description of a synthetic environment consisting of dozens of sequential states. For example, characters move between rooms, and their locations are recorded at each step. To answer a question, the model must process the entire trajectory and correctly aggregate information across all steps.

The tasks contain sequences of 32, 64, or 128 states. Missing or incorrectly processing even one intermediate step may lead to an incorrect final answer.

MMReD evaluates:

• the ability to aggregate information across the entire context;
• robustness when working with long sequences;
• basic calculations without external tools;
• the logical consistency of multi-step reasoning.

The dataset is fully synthetic and procedurally generated. This provides precise ground-truth labels, prevents the tasks from appearing in training corpora, and makes it possible to control the level of difficulty.

The benchmark was developed by the AIRI team and presented in the MMReD research paper at ICLR 2026.

ruAIME

ruAIME is a Russian-language dataset of olympiad-level mathematical problems based on the American Invitational Mathematics Examination format. It includes 724 AIME problems from 1983 to 2025.

The problem statements are usually relatively short, but solving them requires a model to identify hidden structure, find an appropriate transformation, or develop a non-standard strategy.

The dataset covers algebra, geometry, combinatorics, number theory, probability, statistics, and other areas. Performance is evaluated using exact match: the model must produce the correct final answer, with no partial credit awarded for the reasoning process.

The large size of the dataset enables more robust comparison of strong models, as an individual success or failure has less impact on the overall result.

The dataset was created by Vladislav Savenkov, Anton Gabov, and Doubletapp.

What Leaderboard Users Can Access

The leaderboard provides more than an overall model ranking. Users can compare model performance on each dataset separately, analyze their strengths and weaknesses, and track the results of newly released models.

MERA’s open infrastructure allows users to:

• reproduce published results;
• evaluate their own models;
• compare them with models already included in the leaderboard;
• use a ready-made evaluation system without building the entire infrastructure from scratch.

The project aims to make the comparison of models capable of complex reasoning in Russian transparent, reproducible, and accessible to researchers, developers, and companies.

An Open Platform for New Problem Sets

The project was developed with the participation of partners and members of the AI Alliance. Teams from participating organizations contributed to the creation and development of the datasets, while MERA brought them together within a shared open evaluation infrastructure.

The launch of the leaderboard is the first part of a major MERA TEXT update. The platform will continue to expand together with the research community.

Developers of Russian-language mathematical and logical datasets will be able to integrate their own problem sets into the public leaderboard and compare models within a unified evaluation system. This will help create an open ecosystem of Russian-language benchmarks in which new datasets become part of a shared reproducible infrastructure rather than existing in isolation.

The next MERA update will expand both the public and private parts of the platform and introduce new evaluation areas, datasets, and tools for analyzing modern models.

Project repository:https://github.com/MERA-Evaluation/MERA/tree/v2_dev

Instructions for running evaluations and submitting results:https://github.com/MERA-Evaluation/MERA/tree/v2_dev#mera-open-reasoning-leaderboard

Public leaderboard:https://huggingface.co/spaces/MERA-evaluation/MERA_Reason