Meta-Llama-3-70B

MERA Created at 20.09.2024 15:40
0.56
The overall result
100
Place in the rating
Weak tasks:
62
RWSD
70
PARus
109
RCB
213
ruEthics
175
MultiQ
104
ruWorldTree
101
ruOpenBookQA
51
CheGeKa
108
ruMMLU
148
ruHateSpeech
326
ruDetox
256
ruHHH
144
ruTiE
110
ruHumanEval
127
USE
140
MathLogicQA
135
ruMultiAr
89
SimpleAr
176
LCS
150
BPS
57
ruModAr
95
MaMuRAMu
199
ruCodeEval
+19
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.13 Accuracy
RCB 0.566 / 0.526 Accuracy F1 macro
USE 0.262 Grade norm
RWSD 0.642 Accuracy
PARus 0.92 Accuracy
ruTiE 0.768 Accuracy
MultiQ 0.445 / 0.304 F1 Exact match
CheGeKa 0.397 / 0.308 F1 Exact match
ruModAr 0.829 Exact match
MaMuRAMu 0.802 Accuracy
ruMultiAr 0.33 Exact match
ruCodeEval 0.04 / 0.115 / 0.159 Pass@k
MathLogicQA 0.48 Accuracy
ruWorldTree 0.966 / 0.966 Accuracy F1 macro
ruOpenBookQA 0.9 / 0.9 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.966 Accuracy
ruMMLU 0.704 Accuracy
SimpleAr 0.991 Exact match
ruHumanEval 0.187 / 0.42 / 0.512 Pass@k
ruHHH 0.669
ruHateSpeech 0.785
ruDetox 0.145
ruEthics
Correct God Ethical
Virtue 0.259 0.297 0.336
Law 0.288 0.31 0.325
Moral 0.317 0.32 0.37
Justice 0.301 0.314 0.316
Utilitarianism 0.26 0.284 0.316

Information about the submission

Mera version
v.1.2.0
Torch Version
2.3.1
The version of the codebase
6ec745e0
CUDA version
12.1
Precision of the model weights
bfloat16
Seed
1234
Batch
1
Transformers version
4.43.1
The number of GPUs and their type
8 x NVIDIA H100 80GB HBM3
Architecture
hf

Team:

MERA

Name of the ML model:

Meta-Llama-3-70B

Model size

70.6B

Model type:

Opened

Pretrain

Additional links:

https://ai.meta.com/research/publications/the-llama-3-herd-of-models

Architecture description:

An auto-regressive 70B language model that uses an optimized transformer architecture, with context length of 8K tokens. The model uses Grouped-Query Attention.

Description of the training:

Training Time is 6.4M GPU hours on hardware of type H100-80GB.

Pretrain data:

Llama 3 was pretrained on over 15 trillion tokens. The pretraining data includes data from publicly available sources and has a cutoff of December 2023.

License:

https://llama.meta.com/llama3/license

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, rutie, ruhumaneval, rucodeeval - 8192

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
Meta-Llama-3-70B
MERA
0.433 0.433 0.8 0.167 0.1 0.433 0 - 0.033 0 0 0.033 0.333 0.133 0.1 0.367 0.033 0 0 0 0 0.667 0.233 0.1 0.233 0.55 0.167 0.267 0.433 0.367 0.467
Model, team Honest Helpful Harmless
Meta-Llama-3-70B
MERA
0.672 0.695 0.638
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Meta-Llama-3-70B
MERA
0.674 0.524 0.809 0.846 0.843 0.881 0.845 0.804 0.858 0.731 0.553 0.532 0.55 0.796 0.839 0.766 0.82 0.868 0.478 0.763 0.377 0.871 0.4 0.711 0.607 0.83 0.538 0.667 0.763 0.51 0.79 0.843 0.736 0.919 0.74 0.739 0.45 0.861 0.53 0.67 0.899 0.831 0.669 0.546 0.89 0.616 0.907 0.433 0.507 0.757 0.67 0.882 0.774 0.832 0.87 0.836 0.86
Model, team SIM FL STA
Meta-Llama-3-70B
MERA
0.862 0.587 0.284
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
Meta-Llama-3-70B
MERA
0.667 0.881 0.75 0.694 0.829 0.81 0.672 0.702 0.904 0.815 0.821 0.742 0.592 0.76 0.819 0.704 0.738 0.8 0.702 0.877 0.719 0.864 0.844 0.834 0.844 0.864 0.782 0.737 0.93 0.822 0.867 0.846 0.795 0.93 0.712 0.786 0.822 0.889 0.614 0.785 0.898 0.889 0.867 0.778 0.897 0.844 0.897 0.864 0.785 0.947 0.844 0.928 0.823 0.701 0.698 0.766 0.889
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Meta-Llama-3-70B
MERA
0.259 0.288 0.317 0.301 0.26
Model, team Virtue Law Moral Justice Utilitarianism
Meta-Llama-3-70B
MERA
0.297 0.31 0.32 0.314 0.284
Model, team Virtue Law Moral Justice Utilitarianism
Meta-Llama-3-70B
MERA
0.336 0.325 0.37 0.316 0.316
Model, team Women Men LGBT Nationalities Migrants Other
Meta-Llama-3-70B
MERA
0.787 0.714 0.765 0.73 0.714 0.869