Gamayun-1.5B

Gamayun Team Created at 30.10.2025 19:39
0.381
The overall result
355
Place in the rating
Weak tasks:
222
RWSD
341
PARus
325
RCB
457
ruEthics
384
MultiQ
410
ruWorldTree
335
ruOpenBookQA
347
CheGeKa
416
ruMMLU
390
ruHateSpeech
344
ruDetox
491
ruHHH
457
ruTiE
131
ruHumanEval
407
USE
410
MathLogicQA
424
ruMultiAr
429
SimpleAr
272
LCS
597
BPS
336
ruModAr
384
MaMuRAMu
149
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.11 Accuracy
RCB 0.509 / 0.481 Accuracy F1 macro
USE 0.077 Grade norm
RWSD 0.554 Accuracy
PARus 0.708 Accuracy
ruTiE 0.516 Accuracy
MultiQ 0.275 / 0.122 F1 Exact match
CheGeKa 0.059 / 0.034 F1 Exact match
ruModAr 0.409 Exact match
MaMuRAMu 0.517 Accuracy
ruMultiAr 0.146 Exact match
ruCodeEval 0.099 / 0.23 / 0.305 Pass@k
MathLogicQA 0.338 Accuracy
ruWorldTree 0.701 / 0.7 Accuracy F1 macro
ruOpenBookQA 0.685 / 0.685 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.414 Accuracy
ruMMLU 0.425 Accuracy
SimpleAr 0.789 Exact match
ruHumanEval 0.157 / 0.272 / 0.311 Pass@k
ruHHH 0.489
ruHateSpeech 0.592
ruDetox 0.138
ruEthics
Correct God Ethical
Virtue 0.105 0.203 0.137
Law 0.095 0.21 0.156
Moral 0.109 0.203 0.148
Justice 0.105 0.206 0.128
Utilitarianism 0.057 0.165 0.08

Information about the submission

Mera version
v1.2.0
Torch Version
2.4.1
The version of the codebase
fea61e4
CUDA version
11.8
Precision of the model weights
float32
Seed
1234
Batch
1
Transformers version
4.55.0
The number of GPUs and their type
1 x Tesla V100-PCIE-32GB
Architecture
hf

Team:

Gamayun Team

Name of the ML model:

Gamayun-1.5B

Model size

1.5B

Model type:

Closed

SFT

Additional links:

https://www.arxiv.org/abs/2512.21580

Architecture description:

We present Gamayun, a 1.5B-parameter multilingual language model trained entirely from scratch on 2.5T tokens. Designed for efficiency and deployment in resource-constrained environments, Gamayun addresses the lack of research on small non-English-centric LLMs by adopting a novel two-stage pre-training strategy: balanced multilingual training for cross-lingual alignment, followed by high-quality English enrichment to transfer performance gains across languages. Our model supports 12 languages, with special focus on Russian. Despite a significantly smaller training budget than comparable models, Gamayun outperforms LLaMA3.2-1B (9T tokens) on all considered benchmarks, and surpasses Qwen2.5-1.5B (18T tokens) on a wide range of English and multilingual tasks. It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM, achieving state-of-the-art results in Russian, including the MERA benchmark, among the models of comparable size (1-2B parameters).

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=[" \n"]; \nchegeka - do_sample=false;until=[" \n"]; \nrudetox - do_sample=false;until=[" \n"]; \nrumultiar - do_sample=false;until=[" \n"]; \nuse - do_sample=false;until=[" \n","."]; \nmultiq - do_sample=false;until=[" \n"]; \nrumodar - do_sample=false;until=[" \n"]; \nruhumaneval - do_sample=true;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"];temperature=0.6;

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Gamayun-1.5B
Gamayun Team
0.067 0.067 0.467 0.233 0.033 0.133 0 - 0 0.067 0 0.033 0.033 0 0.067 0.267 0.033 0 0.033 0.033 0 0 0 0 0.067 0.075 0.1 0.133 0.033 0.033 0.133
Model, team Honest Helpful Harmless
Gamayun-1.5B
Gamayun Team
0.459 0.475 0.534
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Gamayun-1.5B
Gamayun Team
0.43 0.392 0.493 0.675 0.49 0.532 0.592 0.479 0.444 0.422 0.211 0.278 0.29 0.472 0.53 0.483 0.41 0.34 0.244 0.527 0.25 0.55 0.28 0.457 0.205 0.47 0.331 0.454 0.465 0.36 0.61 0.603 0.442 0.586 0.472 0.325 0.28 0.471 0.377 0.399 0.581 0.375 0.476 0.321 0.562 0.398 0.525 0.307 0.376 0.393 0.35 0.612 0.415 0.466 0.43 0.6 0.477
Model, team SIM FL STA
Gamayun-1.5B
Gamayun Team
0.462 0.697 0.594
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Gamayun-1.5B
Gamayun Team
0.511 0.545 0.367 0.472 0.539 0.414 0.448 0.579 0.519 0.446 0.59 0.5 0.35 0.55 0.474 0.42 0.561 0.444 0.386 0.632 0.263 0.525 0.356 0.604 0.556 0.545 0.526 0.421 0.789 0.533 0.8 0.577 0.429 0.491 0.47 0.5 0.4 0.756 0.421 0.462 0.637 0.571 0.578 0.511 0.793 0.622 0.69 0.5 0.492 0.667 0.533 0.464 0.557 0.519 0.279 0.357 0.6
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Gamayun-1.5B
Gamayun Team
0.105 0.095 0.109 0.105 0.057
Model, team Virtue Law Moral Justice Utilitarianism
Gamayun-1.5B
Gamayun Team
0.203 0.21 0.203 0.206 0.165
Model, team Virtue Law Moral Justice Utilitarianism
Gamayun-1.5B
Gamayun Team
0.137 0.156 0.148 0.128 0.08
Model, team Women Men LGBT Nationalities Migrants Other
Gamayun-1.5B
Gamayun Team
0.63 0.543 0.471 0.703 0.286 0.557