Zero-Mistral-Small-24B-Instruct-2501

ZeroAgency Created at 16.02.2025 08:21
0.518
The overall result
128
Place in the rating
Weak tasks:
184
RWSD
182
PARus
236
RCB
158
ruEthics
204
MultiQ
113
ruWorldTree
119
ruOpenBookQA
128
CheGeKa
136
ruMMLU
83
ruHateSpeech
240
ruDetox
334
ruHHH
281
ruTiE
173
ruHumanEval
114
USE
149
MathLogicQA
214
ruMultiAr
152
SimpleAr
579
LCS
180
BPS
92
ruModAr
119
MaMuRAMu
196
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.03 Accuracy
RCB 0.534 / 0.495 Accuracy F1 macro
USE 0.285 Grade norm
RWSD 0.565 Accuracy
PARus 0.864 Accuracy
ruTiE 0.652 Accuracy
MultiQ 0.414 / 0.289 F1 Exact match
CheGeKa 0.297 / 0.219 F1 Exact match
ruModAr 0.708 Exact match
MaMuRAMu 0.773 Accuracy
ruMultiAr 0.286 Exact match
ruCodeEval 0.043 / 0.161 / 0.25 Pass@k
MathLogicQA 0.476 Accuracy
ruWorldTree 0.962 / 0.962 Accuracy F1 macro
ruOpenBookQA 0.885 / 0.886 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.961 Accuracy
ruMMLU 0.663 Accuracy
SimpleAr 0.982 Exact match
ruHumanEval 0.09 / 0.276 / 0.39 Pass@k
ruHHH 0.601
ruHateSpeech 0.823
ruDetox 0.184
ruEthics
Correct God Ethical
Virtue 0.316 0.424 0.54
Law 0.373 0.439 0.53
Moral 0.362 0.457 0.549
Justice 0.334 0.398 0.488
Utilitarianism 0.295 0.373 0.461

Information about the submission

Mera version
v.1.2.0
Torch Version
2.5.1
The version of the codebase
30667dc
CUDA version
12.4
Precision of the model weights
-
Seed
1234
Batch
1
Transformers version
4.48.3
The number of GPUs and their type
1 x NVIDIA H100 PCIe
Architecture
vllm

Team:

ZeroAgency

Name of the ML model:

Zero-Mistral-Small-24B-Instruct-2501

Model size

24

Model type:

Opened

SFT

Architecture description:

Zero-Mistral-Small is an improved version of mistralai/Mistral-Small-24B-Instruct-2501, primarily adapted for Russian and English languages. The training involved SFT stage on GrandMaster-PRO-MAX dataset.

Description of the training:

LoRa finetune with 96 rank

Pretrain data:

Vikhrmodels/GrandMaster-PRO-MAX dataset

License:

MIT

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, ruhumaneval, rucodeeval, rutie - 4096

System prompt:
You are Zero-Mistral-Small-24B-Instruct-2501, a Large Language Model (LLM) created by ZeroAgency.ru, a Russian startup headquartered in Tambov. \n \nYour knowledge base was last updated on 2023-10-01. \nThe current date is {today}. \n \nWhen you're not sure about some information, you say that you don't have the information and don't make up anything. \nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. "What are some good restaurants around me?" => "Where are you?" or "When is the next flight to Tokyo" => "Where do you travel from?"). \nYou are always very attentive to dates, in particular you try to resolve dates (e.g. "yesterday" is {yesterday}) and when asked about information at specific dates, you discard information that is at another date. \nYou follow these instructions in all languages, and always respond to the user in the language they use or request.

Description of the template:
{%- set today = strftime_now("%Y-%m-%d") %} \n{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \"What are some good restaurants around me?\" => \"Where are you?\" or \"When is the next flight to Tokyo\" => \"Where do you travel from?\")" %} \n \n{{- bos_token }} \n \n{%- if messages[0]['role'] == 'system' %} \n {%- set system_message = messages[0]['content'] %} \n {%- set loop_messages = messages[1:] %} \n{%- else %} \n {%- set system_message = default_system_message %} \n {%- set loop_messages = messages %} \n{%- endif %} \n{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }} \n \n{%- for message in loop_messages %} \n {%- if message['role'] == 'user' %} \n {{- '[INST]' + message['content'] + '[/INST]' }} \n {%- elif message['role'] == 'system' %} \n {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }} \n {%- elif message['role'] == 'assistant' %} \n {{- message['content'] + eos_token }} \n {%- else %} \n {{- raise_exception('Only user, system and assistant roles are supported!') }} \n {%- endif %} \n{%- endfor %}

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.667 0.4 0.833 0.233 0.167 0.567 0.033 - 0 0.033 0 0 0.433 0.033 0.1 0.267 0.033 0.033 0 0.033 0.1 0.767 0.267 0.2 0.133 0.592 0.1 0.3 0.633 0.3 0.4
Model, team Honest Helpful Harmless
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.557 0.61 0.638
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.681 0.53 0.829 0.846 0.788 0.811 0.777 0.72 0.765 0.704 0.474 0.532 0.44 0.741 0.799 0.72 0.69 0.799 0.456 0.733 0.361 0.813 0.45 0.642 0.527 0.85 0.458 0.648 0.767 0.47 0.73 0.826 0.681 0.838 0.74 0.709 0.46 0.845 0.523 0.631 0.808 0.717 0.607 0.629 0.846 0.583 0.877 0.422 0.461 0.685 0.69 0.835 0.738 0.807 0.79 0.764 0.85
Model, team SIM FL STA
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.75 0.621 0.451
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.622 0.861 0.683 0.667 0.816 0.879 0.759 0.719 0.769 0.769 0.769 0.742 0.583 0.744 0.749 0.654 0.785 0.8 0.719 0.807 0.667 0.831 0.8 0.811 0.756 0.833 0.795 0.667 0.912 0.778 0.8 0.872 0.679 0.93 0.636 0.804 0.844 0.822 0.614 0.692 0.833 0.873 0.867 0.889 0.914 0.867 0.897 0.864 0.769 0.912 0.867 0.812 0.785 0.662 0.581 0.69 0.889
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.316 0.373 0.362 0.334 0.295
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.424 0.439 0.457 0.398 0.373
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.54 0.53 0.549 0.488 0.461
Model, team Women Men LGBT Nationalities Migrants Other
Zero-Mistral-Small-24B-Instruct-2501
ZeroAgency
0.833 0.771 0.765 0.784 0.714 0.885