Mistral NeMo

llmarena.ru Created at 21.03.2025 22:00
0.42
The overall result
298
Place in the rating
Weak tasks:
568
RWSD
255
PARus
373
RCB
255
ruEthics
205
MultiQ
254
ruWorldTree
260
ruOpenBookQA
160
CheGeKa
269
ruMMLU
153
ruHateSpeech
135
ruDetox
253
ruHHH
255
ruTiE
259
ruHumanEval
268
USE
383
MathLogicQA
305
ruMultiAr
239
SimpleAr
128
LCS
303
BPS
335
ruModAr
254
MaMuRAMu
289
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.146 Accuracy
RCB 0.491 / 0.334 Accuracy F1 macro
USE 0.153 Grade norm
RWSD 0.423 Accuracy
PARus 0.808 Accuracy
ruTiE 0.675 Accuracy
MultiQ 0.412 / 0.281 F1 Exact match
CheGeKa 0.192 / 0.151 F1 Exact match
ruModAr 0.409 Exact match
MaMuRAMu 0.676 Accuracy
ruMultiAr 0.242 Exact match
ruCodeEval 0.001 / 0.003 / 0.006 Pass@k
MathLogicQA 0.353 Accuracy
ruWorldTree 0.876 / 0.702 Accuracy F1 macro
ruOpenBookQA 0.775 / 0.62 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.909 Accuracy
ruMMLU 0.544 Accuracy
SimpleAr 0.965 Exact match
ruHumanEval 0.012 / 0.012 / 0.012 Pass@k
ruHHH 0.674
ruHateSpeech 0.785
ruDetox 0.251
ruEthics
Correct God Ethical
Virtue 0.306 0.352 0.245
Law 0.342 0.324 0.225
Moral 0.348 0.358 0.256
Justice 0.278 0.3 0.216
Utilitarianism 0.264 0.339 0.246

Information about the submission

Mera version
v.1.2.0
Torch Version
2.5.1
The version of the codebase
30667dc
CUDA version
12.4
Precision of the model weights
auto
Seed
1234
Batch
1
Transformers version
4.49.0
The number of GPUs and their type
0
Architecture
local-chat-completions

Team:

llmarena.ru

Name of the ML model:

Mistral NeMo

Link to the ML model:

https://mistral.ai/news/mistral-nemo

Model size

12.0B

Model type:

API

Opened

SFT

Additional links:

https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407

License:

Apache 2.0

Inference parameters

Generation Parameters:
rucodeeval - do_sample=true;temperature=0.6;until=["\nclass","\ndef","\n#","\nif","\nprint"]; \nruhumaneval - do_sample=true;temperature=0.6;until=["\nclass","\ndef","\n#","\nif","\nprint"];

System prompt:
Реши задачу по инструкции ниже. Не давай никаких объяснений и пояснений к своему ответу. Не пиши ничего лишнего. Пиши только то, что указано в инструкции. Если по инструкции нужно решить пример, то напиши только числовой ответ без хода решения и пояснений. Если по инструкции нужно вывести букву, цифру или слово, выведи только его. Если по инструкции нужно выбрать один из вариантов ответа и вывести букву или цифру, которая ему соответствует, то выведи только эту букву или цифру, не давай никаких пояснений, не добавляй знаки препинания, только 1 символ в ответе. Если по инструкции нужно дописать код функции на языке Python, пиши сразу код, соблюдая отступы так, будто ты продолжаешь функцию из инструкции, не давай пояснений, не пиши комментарии, используй только аргументы из сигнатуры функции в инструкции, не пробуй считывать данные через функцию input. Не извиняйся, не строй диалог. Выдавай только ответ и ничего больше.

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Mistral NeMo
llmarena.ru
0.067 0.133 0.733 0.133 0 0.233 0 - 0.033 0.067 0.033 0 0.267 0.033 0 0.25 0.033 0.067 0 0.033 0.033 0.433 0.033 0.167 0.167 0.208 0.133 0.233 0.333 0.267 0.2
Model, team Honest Helpful Harmless
Mistral NeMo
llmarena.ru
0.639 0.695 0.69
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Mistral NeMo
llmarena.ru
0.548 0.446 0.599 0.748 0.572 0.716 0.68 0.63 0.602 0.601 0.439 0.468 0.43 0.639 0.699 0.564 0.58 0.632 0.333 0.641 0.27 0.731 0.25 0.584 0.348 0.57 0.403 0.611 0.661 0.43 0.6 0.694 0.571 0.727 0.589 0.534 0.38 0.729 0.364 0.448 0.667 0.603 0.49 0.456 0.736 0.468 0.735 0.322 0.365 0.515 0.39 0.781 0.569 0.597 0.68 0.739 0.684
Model, team SIM FL STA
Mistral NeMo
llmarena.ru
0.652 0.709 0.599
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Mistral NeMo
llmarena.ru
0.378 0.772 0.583 0.648 0.776 0.69 0.534 0.632 0.654 0.615 0.654 0.658 0.525 0.698 0.614 0.556 0.692 0.689 0.421 0.719 0.333 0.695 0.622 0.722 0.689 0.667 0.692 0.719 0.912 0.711 0.8 0.782 0.554 0.807 0.621 0.75 0.511 0.756 0.491 0.646 0.8 0.794 0.733 0.578 0.793 0.8 0.828 0.727 0.785 0.877 0.689 0.71 0.835 0.623 0.558 0.561 0.733
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Mistral NeMo
llmarena.ru
0.306 0.342 0.348 0.278 0.264
Model, team Virtue Law Moral Justice Utilitarianism
Mistral NeMo
llmarena.ru
0.352 0.324 0.358 0.3 0.339
Model, team Virtue Law Moral Justice Utilitarianism
Mistral NeMo
llmarena.ru
0.245 0.225 0.256 0.216 0.246
Model, team Women Men LGBT Nationalities Migrants Other
Mistral NeMo
llmarena.ru
0.833 0.686 0.824 0.811 0.571 0.754