Supernova-N1-32B-Preview

Nova Created at 19.02.2025 14:43
0.601
The overall result
64
Place in the rating
Weak tasks:
61
RWSD
26
PARus
38
RCB
235
ruEthics
86
MultiQ
83
ruWorldTree
105
ruOpenBookQA
195
CheGeKa
85
ruMMLU
136
ruHateSpeech
78
ruDetox
116
ruHHH
76
ruTiE
115
USE
77
MathLogicQA
90
ruMultiAr
174
SimpleAr
213
LCS
137
ruModAr
88
MaMuRAMu
+16
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.12 Accuracy
RCB 0.591 / 0.554 Accuracy F1 macro
USE 0.283 Grade norm
RWSD 0.642 Accuracy
PARus 0.938 Accuracy
ruTiE 0.847 Accuracy
MultiQ 0.538 / 0.388 F1 Exact match
CheGeKa 0.17 / 0.125 F1 Exact match
ruModAr 0.612 Exact match
MaMuRAMu 0.809 Accuracy
ruMultiAr 0.37 Exact match
ruCodeEval 0.677 / 0.677 / 0.677 Pass@k
MathLogicQA 0.663 Accuracy
ruWorldTree 0.973 / 0.973 Accuracy F1 macro
ruOpenBookQA 0.898 / 0.898 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.999 Accuracy
ruMMLU 0.737 Accuracy
SimpleAr 0.979 Exact match
ruHumanEval 0.671 / 0.671 / 0.671 Pass@k
ruHHH 0.815
ruHateSpeech 0.789
ruDetox 0.313
ruEthics
Correct God Ethical
Virtue 0.296 0.304 0.322
Law 0.3 0.289 0.304
Moral 0.311 0.318 0.337
Justice 0.277 0.255 0.28
Utilitarianism 0.237 0.258 0.282

Information about the submission

Mera version
v.1.2.0
Torch Version
2.5.1
The version of the codebase
30667dc
CUDA version
12.4
Precision of the model weights
-
Seed
1234
Batch
1
Transformers version
4.49.0
The number of GPUs and their type
8 x NVIDIA H100 80GB HBM3
Architecture
local-chat-completions

Team:

Nova

Name of the ML model:

Supernova-N1-32B-Preview

Model size

32.0B

Model type:

Closed

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

System prompt:
Решай задачу строго по инструкции. Только ответ, без объяснений. Числовой ответ — только число. Буква, цифра или слово — только их. Выбор варианта ответа — одна буква или цифра. Ответ должен быть точным, без лишних символов или слов. ФОРМАТ ОТВЕТА ДЛЯ PYTHON: 1. Начните с ```python 2. СРАЗУ добавьте 4 пробела 3. Напишите эффективное решение, не повторяй импорты и название функции, только тело функции 4. Закончите ``` - НЕ пишите def/docstring/комментарии - НЕ допускайте строк без 4 пробелов

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Supernova-N1-32B-Preview
Nova
0.5 0.333 0.833 0.133 0.2 0.4 0.2 - 0.033 0.067 0.1 0.033 0.167 0.133 0.133 0.4 0.033 0.033 0.067 0.067 0.133 0.4 0.367 0.2 0.1 0.717 0.033 0.133 0.5 0.333 0.3
Model, team Honest Helpful Harmless
Supernova-N1-32B-Preview
Nova
0.787 0.78 0.879
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Supernova-N1-32B-Preview
Nova
0.667 0.578 0.895 0.842 0.817 0.836 0.786 0.765 0.812 0.767 0.605 0.619 0.54 0.796 0.811 0.772 0.81 0.854 0.656 0.817 0.582 0.877 0.64 0.717 0.688 0.9 0.525 0.63 0.759 0.56 0.8 0.835 0.736 0.889 0.796 0.808 0.66 0.887 0.702 0.709 0.828 0.79 0.71 0.817 0.867 0.764 0.887 0.596 0.567 0.733 0.71 0.873 0.803 0.861 0.86 0.806 0.876
Model, team SIM FL STA
Supernova-N1-32B-Preview
Nova
0.676 0.709 0.693
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Supernova-N1-32B-Preview
Nova
0.622 0.891 0.75 0.685 0.829 0.793 0.69 0.719 0.808 0.785 0.859 0.792 0.583 0.814 0.76 0.753 0.776 0.822 0.772 0.789 0.807 0.881 0.911 0.876 0.844 0.818 0.833 0.702 0.947 0.822 0.844 0.91 0.821 0.912 0.727 0.857 0.867 0.8 0.772 0.754 0.849 0.841 0.822 0.956 0.879 0.911 0.914 0.932 0.862 0.895 0.867 0.855 0.835 0.74 0.605 0.737 0.878
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Supernova-N1-32B-Preview
Nova
0.296 0.3 0.311 0.277 0.237
Model, team Virtue Law Moral Justice Utilitarianism
Supernova-N1-32B-Preview
Nova
0.304 0.289 0.318 0.255 0.258
Model, team Virtue Law Moral Justice Utilitarianism
Supernova-N1-32B-Preview
Nova
0.322 0.304 0.337 0.28 0.282
Model, team Women Men LGBT Nationalities Migrants Other
Supernova-N1-32B-Preview
Nova
0.806 0.686 0.765 0.838 0.714 0.803