Mistral-7B-Instruct-v0.2

MERA Created at 20.09.2024 11:00
0.405
The overall result
319
Place in the rating
Weak tasks:
101
RWSD
398
PARus
359
RCB
327
ruEthics
388
MultiQ
329
ruWorldTree
365
ruOpenBookQA
213
CheGeKa
361
ruMMLU
315
ruHateSpeech
210
ruDetox
337
ruHHH
415
ruTiE
84
ruHumanEval
394
USE
439
MathLogicQA
422
ruMultiAr
374
SimpleAr
432
LCS
307
BPS
323
ruModAr
346
MaMuRAMu
78
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.082 Accuracy
RCB 0.498 / 0.469 Accuracy F1 macro
USE 0.08 Grade norm
RWSD 0.6 Accuracy
PARus 0.642 Accuracy
ruTiE 0.544 Accuracy
MultiQ 0.272 / 0.088 F1 Exact match
CheGeKa 0.147 / 0.096 F1 Exact match
ruModAr 0.433 Exact match
MaMuRAMu 0.562 Accuracy
ruMultiAr 0.151 Exact match
ruCodeEval 0.275 / 0.478 / 0.555 Pass@k
MathLogicQA 0.322 Accuracy
ruWorldTree 0.781 / 0.779 Accuracy F1 macro
ruOpenBookQA 0.653 / 0.65 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.908 Accuracy
ruMMLU 0.457 Accuracy
SimpleAr 0.881 Exact match
ruHumanEval 0.26 / 0.417 / 0.488 Pass@k
ruHHH 0.601
ruHateSpeech 0.657
ruDetox 0.203
ruEthics
Correct God Ethical
Virtue 0.198 0.236 0.273
Law 0.218 0.234 0.277
Moral 0.242 0.249 0.283
Justice 0.165 0.191 0.237
Utilitarianism 0.18 0.173 0.253

Information about the submission

Mera version
v.1.2.0
Torch Version
2.4.0
The version of the codebase
6ec745e0
CUDA version
12.1
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.43.2
The number of GPUs and their type
1 x NVIDIA H100 80GB HBM3
Architecture
hf

Team:

MERA

Name of the ML model:

Mistral-7B-Instruct-v0.2

Model size

7.2B

Model type:

Opened

SFT

Additional links:

https://arxiv.org/abs/2310.06825

Architecture description:

An instruct fine-tuned version of the Mistral-7B-v0.2, which inherits the Mistral-7B-v0.1 architecture with the following differences: No Sliding-Window Attention, Context length expanded to 32K tokens.

Description of the training:

Fine-tuned on instruction datasets publicly available on the Hugging Face repository. No proprietary data or training tricks were utilized.

Pretrain data:

Open Web data (further details are not disclosed by the authors).

License:

apache-2.0

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, rutie, ruhumaneval, rucodeeval - 32768

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Mistral-7B-Instruct-v0.2
MERA
0.233 0.167 0.667 0.133 0 0.067 0 - 0 0 0 0.033 0.067 0 0.067 0.2 0 0 0 0.067 0 0.2 0.033 0 0.033 0 0.067 0.067 0.167 0.033 0.233
Model, team Honest Helpful Harmless
Mistral-7B-Instruct-v0.2
MERA
0.623 0.61 0.569
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Mistral-7B-Instruct-v0.2
MERA
0.437 0.404 0.493 0.679 0.477 0.682 0.592 0.531 0.509 0.502 0.298 0.31 0.3 0.593 0.573 0.549 0.53 0.438 0.356 0.55 0.213 0.602 0.3 0.41 0.339 0.46 0.342 0.537 0.641 0.4 0.57 0.62 0.54 0.677 0.483 0.432 0.33 0.529 0.358 0.424 0.626 0.43 0.455 0.332 0.552 0.384 0.613 0.311 0.358 0.438 0.44 0.671 0.39 0.445 0.57 0.655 0.544
Model, team SIM FL STA
Mistral-7B-Instruct-v0.2
MERA
0.623 0.544 0.655
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Mistral-7B-Instruct-v0.2
MERA
0.444 0.554 0.5 0.463 0.513 0.552 0.414 0.579 0.596 0.6 0.513 0.567 0.425 0.543 0.526 0.605 0.579 0.6 0.316 0.632 0.246 0.627 0.533 0.609 0.511 0.576 0.628 0.579 0.825 0.756 0.644 0.615 0.473 0.702 0.5 0.571 0.422 0.733 0.351 0.538 0.653 0.54 0.778 0.378 0.776 0.689 0.741 0.477 0.554 0.719 0.622 0.536 0.646 0.506 0.419 0.48 0.667
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Mistral-7B-Instruct-v0.2
MERA
0.198 0.218 0.242 0.165 0.18
Model, team Virtue Law Moral Justice Utilitarianism
Mistral-7B-Instruct-v0.2
MERA
0.236 0.234 0.249 0.191 0.173
Model, team Virtue Law Moral Justice Utilitarianism
Mistral-7B-Instruct-v0.2
MERA
0.273 0.277 0.283 0.237 0.253
Model, team Women Men LGBT Nationalities Migrants Other
Mistral-7B-Instruct-v0.2
MERA
0.648 0.686 0.588 0.703 0.429 0.672