Qwen2-1.5B-Instruct

MERA Created at 22.09.2024 21:49
0.338
The overall result
422
Place in the rating
Weak tasks:
391
RWSD
480
PARus
496
RCB
419
ruEthics
403
MultiQ
406
ruWorldTree
385
ruOpenBookQA
400
CheGeKa
403
ruMMLU
422
ruHateSpeech
301
ruDetox
391
ruHHH
391
ruTiE
215
ruHumanEval
507
USE
420
MathLogicQA
434
ruMultiAr
346
SimpleAr
457
LCS
363
BPS
372
ruModAr
406
MaMuRAMu
227
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.076 Accuracy
RCB 0.395 / 0.318 Accuracy F1 macro
USE 0.053 Grade norm
RWSD 0.504 Accuracy
PARus 0.54 Accuracy
ruTiE 0.558 Accuracy
MultiQ 0.265 / 0.137 F1 Exact match
CheGeKa 0.032 / 0.017 F1 Exact match
ruModAr 0.392 Exact match
MaMuRAMu 0.505 Accuracy
ruMultiAr 0.14 Exact match
ruCodeEval 0.013 / 0.048 / 0.067 Pass@k
MathLogicQA 0.334 Accuracy
ruWorldTree 0.707 / 0.705 Accuracy F1 macro
ruOpenBookQA 0.638 / 0.637 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.849 Accuracy
ruMMLU 0.434 Accuracy
SimpleAr 0.908 Exact match
ruHumanEval 0.043 / 0.115 / 0.152 Pass@k
ruHHH 0.551
ruHateSpeech 0.558
ruDetox 0.159
ruEthics
Correct God Ethical
Virtue 0.171 0.193 0.14
Law 0.151 0.192 0.135
Moral 0.185 0.202 0.174
Justice 0.118 0.153 0.091
Utilitarianism 0.13 0.168 0.109

Information about the submission

Mera version
v.1.2.0
Torch Version
2.4.0
The version of the codebase
6ec745e0
CUDA version
12.1
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.43.2
The number of GPUs and their type
1 x NVIDIA H100 80GB HBM3
Architecture
hf

Team:

MERA

Name of the ML model:

Qwen2-1.5B-Instruct

Model size

1.5B

Model type:

Opened

SFT

Additional links:

https://arxiv.org/pdf/2407.10671

Architecture description:

Qwen2-1.5B-instruct is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, RoPE and RMSNorm

Description of the training:

SFT and RLHF on the mix of manually and synthetically annotated samples after pertaining with next-token prediction task.

Pretrain data:

The post-training data primarily consists of two components: demonstration data D = {($x_i$ , $y_i$ )} and preference data P = {($x_i$ , $y_i^+$, $y_i^-$ )}, where $x_i$ represents the instruction, $y_i$ represents a satisfactory response, and $y_i^+$ and $y_i^-$ are two responses to $x_i$, with $y_i^+$ being the preferred choice over $y_i^-$. The set D is utilized in SFT, whereas P is employed in RLHF. We have assembled an extensive instruction dataset featuring more than 500,000 examples that cover skills such as instruction following, coding, mathematics, logical reasoning, role-playing, multilingualism, and safety.

License:

Apache-2.0

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, rutie, ruhumaneval, rucodeeval - 32768

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Qwen2-1.5B-Instruct
MERA
0.033 0.1 0.333 0.2 0 0.1 0 - 0 0 0 0 0.033 0 0.133 0.133 0.033 0 0 0.033 0 0 0 0 0.033 0.067 0.033 0.033 0.067 0.033 0.067
Model, team Honest Helpful Harmless
Qwen2-1.5B-Instruct
MERA
0.541 0.542 0.569
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Qwen2-1.5B-Instruct
MERA
0.459 0.41 0.441 0.679 0.516 0.637 0.592 0.476 0.438 0.466 0.281 0.246 0.35 0.528 0.501 0.465 0.48 0.354 0.322 0.496 0.25 0.544 0.39 0.405 0.366 0.45 0.352 0.435 0.588 0.37 0.58 0.678 0.479 0.626 0.483 0.402 0.28 0.474 0.318 0.369 0.586 0.342 0.49 0.39 0.547 0.343 0.5 0.352 0.348 0.387 0.42 0.561 0.472 0.416 0.45 0.588 0.425
Model, team SIM FL STA
Qwen2-1.5B-Instruct
MERA
0.588 0.643 0.522
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Qwen2-1.5B-Instruct
MERA
0.444 0.525 0.3 0.565 0.434 0.517 0.483 0.386 0.404 0.477 0.603 0.533 0.392 0.605 0.38 0.481 0.598 0.444 0.474 0.579 0.263 0.508 0.378 0.479 0.6 0.545 0.615 0.526 0.737 0.556 0.756 0.603 0.473 0.456 0.409 0.464 0.444 0.644 0.316 0.523 0.494 0.492 0.644 0.489 0.776 0.622 0.517 0.295 0.662 0.684 0.578 0.406 0.684 0.494 0.349 0.404 0.556
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Qwen2-1.5B-Instruct
MERA
0.171 0.151 0.185 0.118 0.13
Model, team Virtue Law Moral Justice Utilitarianism
Qwen2-1.5B-Instruct
MERA
0.193 0.192 0.202 0.153 0.168
Model, team Virtue Law Moral Justice Utilitarianism
Qwen2-1.5B-Instruct
MERA
0.14 0.135 0.174 0.091 0.109
Model, team Women Men LGBT Nationalities Migrants Other
Qwen2-1.5B-Instruct
MERA
0.537 0.571 0.471 0.595 0.143 0.639