DeepSeek-R1-Distill-Qwen-14B

НГУ Created at 28.04.2025 07:08
0.446
The overall result
247
Place in the rating
Weak tasks:
293
RWSD
283
PARus
227
RCB
239
ruEthics
463
MultiQ
194
ruWorldTree
213
ruOpenBookQA
288
CheGeKa
186
ruMMLU
370
ruHateSpeech
149
ruDetox
223
ruHHH
371
ruTiE
258
ruHumanEval
180
USE
115
MathLogicQA
367
ruMultiAr
272
SimpleAr
313
LCS
203
BPS
158
ruModAr
159
MaMuRAMu
261
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.1 Accuracy
RCB 0.537 / 0.485 Accuracy F1 macro
USE 0.199 Grade norm
RWSD 0.527 Accuracy
PARus 0.766 Accuracy
ruTiE 0.572 Accuracy
MultiQ 0.226 / 0.107 F1 Exact match
CheGeKa 0.1 / 0.072 F1 Exact match
ruModAr 0.55 Exact match
MaMuRAMu 0.74 Accuracy
ruMultiAr 0.19 Exact match
ruCodeEval 0.002 / 0.009 / 0.018 Pass@k
MathLogicQA 0.52 Accuracy
ruWorldTree 0.931 / 0.931 Accuracy F1 macro
ruOpenBookQA 0.828 / 0.827 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.956 Accuracy
ruMMLU 0.629 Accuracy
SimpleAr 0.945 Exact match
ruHumanEval 0.012 / 0.049 / 0.085 Pass@k
ruHHH 0.713
ruHateSpeech 0.611
ruDetox 0.236
ruEthics
Correct God Ethical
Virtue 0.307 0.346 0.371
Law 0.319 0.34 0.359
Moral 0.34 0.367 0.389
Justice 0.265 0.277 0.333
Utilitarianism 0.23 0.281 0.333

Information about the submission

Mera version
v.1.2.0
Torch Version
2.6.0
The version of the codebase
9e50515
CUDA version
12.4
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.51.1
The number of GPUs and their type
1 x NVIDIA A100 80GB PCIe
Architecture
hf

Team:

НГУ

Name of the ML model:

DeepSeek-R1-Distill-Qwen-14B

Model size

14.8B

Model type:

Opened

SFT

Architecture description:

Type: Causal Language Models Training Stage: Pretraining & Post-training Architecture: transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias

Description of the training:

DeepSeek-R1-Distill-Qwen-14B is derived from Qwen-2.5 series, which are originally licensed under Apache 2.0 License, and now finetuned with 800k samples curated with DeepSeek-R1.

Pretrain data:

The model was pretrained with a large amount of data of English, Chinese and over 27 additional languages including Russian. In terms of the context length, the model was pretrained on data of the context length of 32K tokens.

License:

MIT License

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;temperature=0.6;until=["\nclass","\ndef","\n#","\nif","\nprint"]; \nrucodeeval - do_sample=true;temperature=0.6;until=["\nclass","\ndef","\n#","\nif","\nprint"];

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, ruhumaneval, rucodeeval, rutie - 131072

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.467 0.233 0.767 0.167 0 0.167 0.033 - 0.033 0.033 0.033 0 0.133 0.067 0.1 0.233 0.1 0 0 0 0.033 0.567 0.2 0.133 0.1 0.433 0.233 0.267 0.367 0.133 0.2
Model, team Honest Helpful Harmless
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.77 0.627 0.741
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.556 0.5 0.816 0.795 0.68 0.751 0.777 0.672 0.667 0.677 0.5 0.571 0.43 0.657 0.719 0.636 0.7 0.729 0.544 0.702 0.469 0.749 0.43 0.613 0.491 0.66 0.436 0.574 0.71 0.48 0.71 0.752 0.644 0.818 0.634 0.709 0.52 0.803 0.563 0.557 0.793 0.632 0.634 0.637 0.796 0.667 0.77 0.452 0.436 0.618 0.58 0.776 0.687 0.723 0.83 0.733 0.762
Model, team SIM FL STA
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.765 0.623 0.526
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.578 0.743 0.617 0.648 0.803 0.69 0.672 0.684 0.673 0.631 0.756 0.758 0.517 0.76 0.708 0.63 0.757 0.778 0.632 0.789 0.719 0.864 0.733 0.769 0.778 0.818 0.731 0.702 0.877 0.756 0.822 0.821 0.75 0.842 0.636 0.75 0.822 0.778 0.561 0.723 0.824 0.746 0.778 0.822 0.931 0.844 0.776 0.818 0.754 0.86 0.8 0.768 0.848 0.675 0.651 0.626 0.811
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.307 0.319 0.34 0.265 0.23
Model, team Virtue Law Moral Justice Utilitarianism
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.346 0.34 0.367 0.277 0.281
Model, team Virtue Law Moral Justice Utilitarianism
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.371 0.359 0.389 0.333 0.333
Model, team Women Men LGBT Nationalities Migrants Other
DeepSeek-R1-Distill-Qwen-14B
НГУ
0.556 0.629 0.588 0.622 0.286 0.738