gemma-2-9b-it

MERA Created at 22.09.2024 22:16
0.522
The overall result
125
Place in the rating
Weak tasks:
158
RWSD
197
PARus
183
RCB
110
ruEthics
189
MultiQ
144
ruWorldTree
171
ruOpenBookQA
158
CheGeKa
198
ruMMLU
203
ruHateSpeech
214
ruDetox
108
ruHHH
131
ruTiE
103
ruHumanEval
133
USE
145
MathLogicQA
149
ruMultiAr
162
SimpleAr
312
LCS
208
BPS
277
ruModAr
169
MaMuRAMu
97
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.1 Accuracy
RCB 0.548 / 0.493 Accuracy F1 macro
USE 0.242 Grade norm
RWSD 0.573 Accuracy
PARus 0.85 Accuracy
ruTiE 0.778 Accuracy
MultiQ 0.43 / 0.304 F1 Exact match
CheGeKa 0.193 / 0.159 F1 Exact match
ruModAr 0.469 Exact match
MaMuRAMu 0.726 Accuracy
ruMultiAr 0.319 Exact match
ruCodeEval 0.215 / 0.494 / 0.561 Pass@k
MathLogicQA 0.478 Accuracy
ruWorldTree 0.949 / 0.948 Accuracy F1 macro
ruOpenBookQA 0.853 / 0.852 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.955 Accuracy
ruMMLU 0.618 Accuracy
SimpleAr 0.981 Exact match
ruHumanEval 0.193 / 0.453 / 0.561 Pass@k
ruHHH 0.826
ruHateSpeech 0.758
ruDetox 0.2
ruEthics
Correct God Ethical
Virtue 0.382 0.391 0.37
Law 0.366 0.36 0.36
Moral 0.402 0.407 0.387
Justice 0.338 0.334 0.323
Utilitarianism 0.322 0.374 0.32

Information about the submission

Mera version
v.1.2.0
Torch Version
2.4.0
The version of the codebase
6ec745e0
CUDA version
12.1
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.43.2
The number of GPUs and their type
1 x NVIDIA H100 80GB HBM3
Architecture
hf

Team:

MERA

Name of the ML model:

gemma-2-9b-it

Model size

9.2B

Model type:

Opened

SFT

Additional links:

https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf

Architecture description:

Gemma 2 is a text-to-text, decoder-only large language model, available in English. • SentencePiece tokenizer of size 256k

Description of the training:

Fine-tuning pretrained Gemma 2: • Knowledge Distillation • supervised fine-tuning • RLHF

Pretrain data:

Pretraining data: Web Documents, Code, Mathematics Fine-tuning data: the post-training data from Gemma 1.1 with a mixture of internal and external public data

License:

https://ai.google.dev/gemma/terms

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, rutie, ruhumaneval, rucodeeval - 8192

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
gemma-2-9b-it
MERA
0.3 0.367 0.8 0.067 0.067 0.4 0.033 - 0.133 0.033 0 0.033 0.133 0 0.1 0.3 0 0 0 0.033 0.067 0.733 0.167 0.3 0.1 0.575 0.1 0.2 0.4 0.4 0.367
Model, team Honest Helpful Harmless
gemma-2-9b-it
MERA
0.836 0.746 0.897
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
gemma-2-9b-it
MERA
0.578 0.482 0.717 0.782 0.696 0.751 0.757 0.678 0.651 0.646 0.553 0.516 0.47 0.722 0.718 0.685 0.7 0.806 0.433 0.695 0.284 0.76 0.37 0.642 0.473 0.68 0.458 0.657 0.727 0.45 0.73 0.752 0.65 0.828 0.634 0.632 0.38 0.835 0.536 0.626 0.798 0.621 0.621 0.57 0.811 0.574 0.789 0.444 0.422 0.603 0.49 0.806 0.715 0.702 0.77 0.812 0.793
Model, team SIM FL STA
gemma-2-9b-it
MERA
0.357 0.79 0.761
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
gemma-2-9b-it
MERA
0.556 0.762 0.55 0.63 0.724 0.741 0.672 0.596 0.673 0.738 0.718 0.633 0.575 0.76 0.643 0.667 0.71 0.844 0.544 0.789 0.649 0.746 0.822 0.811 0.689 0.788 0.679 0.702 0.947 0.8 0.822 0.782 0.67 0.842 0.682 0.714 0.778 0.867 0.526 0.738 0.804 0.825 0.756 0.778 0.914 0.933 0.862 0.864 0.769 0.895 0.778 0.696 0.797 0.649 0.535 0.596 0.778
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
gemma-2-9b-it
MERA
0.382 0.366 0.402 0.338 0.322
Model, team Virtue Law Moral Justice Utilitarianism
gemma-2-9b-it
MERA
0.391 0.36 0.407 0.334 0.374
Model, team Virtue Law Moral Justice Utilitarianism
gemma-2-9b-it
MERA
0.37 0.36 0.387 0.323 0.32
Model, team Women Men LGBT Nationalities Migrants Other
gemma-2-9b-it
MERA
0.787 0.6 0.765 0.784 0.429 0.82