Zero-Mistral-24B

ZeroAgency Created at 21.04.2025 18:20
0.623
The overall result
52
Place in the rating
In the top by tasks:
6
RCB
The result on the task is higher than human
The task is one of the main ones
2
MultiQ
The task is one of the main ones
7
ruHHH
The result on the task is higher than human
Weak tasks:
228
RWSD
20
PARus
96
ruEthics
42
ruWorldTree
80
ruOpenBookQA
50
CheGeKa
47
ruMMLU
65
ruHateSpeech
38
ruDetox
44
ruTiE
290
ruHumanEval
33
USE
92
MathLogicQA
99
ruMultiAr
42
SimpleAr
47
LCS
97
BPS
26
ruModAr
61
MaMuRAMu
+15
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.194 Accuracy
RCB 0.607 / 0.592 Accuracy F1 macro
USE 0.452 Grade norm
RWSD 0.55 Accuracy
PARus 0.942 Accuracy
ruTiE 0.868 Accuracy
MultiQ 0.781 / 0.629 F1 Exact match
CheGeKa 0.397 / 0.322 F1 Exact match
ruModAr 0.971 Exact match
MaMuRAMu 0.832 Accuracy
ruMultiAr 0.354 Exact match
ruCodeEval 0 / 0 / 0 Pass@k
MathLogicQA 0.613 Accuracy
ruWorldTree 0.987 / 0.987 Accuracy F1 macro
ruOpenBookQA 0.913 / 0.913 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.981 Accuracy
ruMMLU 0.778 Accuracy
SimpleAr 0.997 Exact match
ruHumanEval 0.006 / 0.006 / 0.006 Pass@k
ruHHH 0.916
ruHateSpeech 0.834
ruDetox 0.341
ruEthics
Correct God Ethical
Virtue 0.386 0.421 0.653
Law 0.399 0.427 0.65
Moral 0.41 0.452 0.697
Justice 0.333 0.375 0.596
Utilitarianism 0.327 0.363 0.573

Information about the submission

Mera version
v.1.2.0
Torch Version
2.5.1
The version of the codebase
4ebdda5
CUDA version
12.4
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.51.0
The number of GPUs and their type
8 x NVIDIA H200
Architecture
local-chat-completions

Team:

ZeroAgency

Name of the ML model:

Zero-Mistral-24B

Model size

23.6B

Model type:

Opened

SFT

Architecture description:

Zero-Mistral is an improved TEXT-only version of mistralai/Mistral-Small-3.1-24B-Instruct-2503, primarily adapted for Russian and English languages. The original Mistral model contains vision features which were removed from this model. The training involved SFT stage primarily on Big Russian Dataset dataset and proprietary dataset from Shkolkovo.online.

Description of the training:

LoRa fine-tune with 256 rank.

Pretrain data:

ZeroAgency/ru-big-russian-dataset

License:

MIT

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=[" \n"]; \nchegeka - do_sample=false;until=[" \n"]; \nrudetox - do_sample=false;until=[" \n"]; \nrumultiar - do_sample=false;until=[" \n"]; \nuse - do_sample=false;until=[" \n","."]; \nmultiq - do_sample=false;until=[" \n"]; \nrumodar - do_sample=false;until=[" \n"]; \nruhumaneval - do_sample=true;temperature=0.6;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"]; \nrucodeeval - do_sample=true;temperature=0.6;until=[" \nclass"," \ndef"," \n#"," \nif"," \nprint"];

The size of the context:
128k

System prompt:
Реши задачу по инструкции ниже. Не давай никаких объяснений и пояснений к своему ответу. Не пиши ничего лишнего. Пиши только то, что указано в инструкции. Если по инструкции нужно решить пример, то напиши только числовой ответ без хода решения и пояснений. Если по инструкции нужно вывести букву, цифру или слово, выведи только его. Если по инструкции нужно выбрать один из вариантов ответа и вывести букву или цифру, которая ему соответствует, то выведи только эту букву или цифру, не давай никаких пояснений, не добавляй знаки препинания, только 1 символ в ответе. Если по инструкции нужно дописать код функции на языке Python, пиши сразу код, соблюдая отступы так, будто ты продолжаешь функцию из инструкции, не давай пояснений, не пиши комментарии, используй только аргументы из сигнатуры функции в инструкции, не пробуй считывать данные через функцию input. Не извиняйся, не строй диалог. Выдавай только ответ и ничего больше.

Description of the template:
{%- set today = strftime_now("%Y-%m-%d") %} {%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris. \nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ". \n \nWhen you're not sure about some information, you say that you don't have the information and don't make up anything. \nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. "What are some good restaurants around me?" => "Where are you?" or "When is the next flight to Tokyo" => "Where do you travel from?")" %} {{- bos_token }} {%- if messages[0]['role'] == 'system' %} {%- if messages[0]['content'] is string %} {%- set system_message = messages[0]['content'] %} {%- else %} {%- set system_message = messages[0]['content'][0]['text'] %} {%- endif %} {%- set loop_messages = messages[1:] %} {%- else %} {%- set system_message = default_system_message %} {%- set loop_messages = messages %} {%- endif %} {{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }} {%- for message in loop_messages %} {%- if message['role'] == 'user' %} {%- if message['content'] is string %} {{- '[INST]' + message['content'] + '[/INST]' }} {%- else %} {{- '[INST]' }} {%- for block in message['content'] %} {%- if block['type'] == 'text' %} {{- block['text'] }} {%- elif block['type'] in ['image', 'image_url'] %} {{- '[IMG]' }} {%- else %} {{- raise_exception('Only text and image blocks are supported in message content!') }} {%- endif %} {%- endfor %} {{- '[/INST]' }} {%- endif %} {%- elif message['role'] == 'system' %} {%- if message['content'] is string %} {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }} {%- else %} {{- '[SYSTEM_PROMPT]' + message['content'][0]['text'] + '[/SYSTEM_PROMPT]' }} {%- endif %} {%- elif message['role'] == 'assistant' %} {%- if message['content'] is string %} {{- message['content'] + eos_token }} {%- else %} {{- message['content'][0]['text'] + eos_token }} {%- endif %} {%- else %} {{- raise_exception('Only user, system and assistant roles are supported!') }} {%- endif %} {%- endfor %}

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Zero-Mistral-24B
ZeroAgency
0.867 0.533 0.767 0.133 0.1 0.7 0.233 - 0.267 0.033 0.067 0.033 0.5 0.3 0 0.617 0.167 0.167 0.3 0.133 0.267 0.733 0.567 0.433 0.333 0.758 0.733 0.7 0.767 0.567 0.7
Model, team Honest Helpful Harmless
Zero-Mistral-24B
ZeroAgency
0.902 0.881 0.966
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Zero-Mistral-24B
ZeroAgency
0.726 0.554 0.914 0.893 0.869 0.876 0.845 0.781 0.855 0.758 0.579 0.587 0.66 0.815 0.875 0.757 0.78 0.938 0.667 0.863 0.737 0.901 0.53 0.757 0.688 0.91 0.602 0.704 0.8 0.55 0.84 0.868 0.773 0.889 0.826 0.825 0.61 0.919 0.702 0.768 0.874 0.89 0.724 0.817 0.919 0.736 0.931 0.5 0.606 0.772 0.73 0.882 0.841 0.903 0.91 0.848 0.896
Model, team SIM FL STA
Zero-Mistral-24B
ZeroAgency
0.843 0.624 0.66
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Zero-Mistral-24B
ZeroAgency
0.711 0.911 0.767 0.694 0.908 0.776 0.759 0.737 0.885 0.815 0.808 0.8 0.508 0.86 0.83 0.753 0.785 0.844 0.772 0.895 0.895 0.915 0.911 0.899 0.867 0.939 0.846 0.754 0.93 0.867 0.822 0.897 0.83 0.947 0.727 0.857 0.867 0.889 0.649 0.8 0.894 0.889 0.889 1 0.948 0.911 0.948 0.932 0.877 0.93 0.889 0.913 0.823 0.727 0.558 0.772 0.9
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-24B
ZeroAgency
0.386 0.399 0.41 0.333 0.327
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-24B
ZeroAgency
0.421 0.427 0.452 0.375 0.363
Model, team Virtue Law Moral Justice Utilitarianism
Zero-Mistral-24B
ZeroAgency
0.653 0.65 0.697 0.596 0.573
Model, team Women Men LGBT Nationalities Migrants Other
Zero-Mistral-24B
ZeroAgency
0.87 0.743 0.824 0.703 0.857 0.902