ruT5-base

RussianNLP Created at 20.09.2024 12:58
0.189
The overall result
606
Place in the rating
Weak tasks:
506
RWSD
584
PARus
524
RCB
616
MultiQ
593
ruWorldTree
618
ruOpenBookQA
614
CheGeKa
595
ruMMLU
574
ruHateSpeech
603
ruDetox
488
ruHHH
567
ruTiE
578
MathLogicQA
602
SimpleAr
399
LCS
538
BPS
610
MaMuRAMu
+13
Hide

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.088 Accuracy
RCB 0.37 / 0.277 Accuracy F1 macro
USE 0 Grade norm
RWSD 0.473 Accuracy
PARus 0.48 Accuracy
ruTiE 0.495 Accuracy
MultiQ 0.015 / 0 F1 Exact match
CheGeKa 0.002 / 0 F1 Exact match
ruModAr 0 Exact match
MaMuRAMu 0.25 Accuracy
ruMultiAr 0 Exact match
ruCodeEval 0 / 0 / 0 Pass@k
MathLogicQA 0.253 Accuracy
ruWorldTree 0.272 / 0.223 Accuracy F1 macro
ruOpenBookQA 0.243 / 0.19 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.513 Accuracy
ruMMLU 0.256 Accuracy
SimpleAr 0.001 Exact match
ruHumanEval 0 / 0 / 0 Pass@k
ruHHH 0.494
ruHateSpeech 0.468
ruDetox 0.045
ruEthics
Correct God Ethical
Virtue -0.029 -0.062 -0.101
Law -0.016 -0.04 -0.103
Moral -0.007 -0.038 -0.079
Justice -0.004 -0.029 -0.081
Utilitarianism -0.042 -0.045 -0.11

Information about the submission

Mera version
v.1.2.0
Torch Version
2.3.1
The version of the codebase
aec92f8
CUDA version
12.1
Precision of the model weights
float16
Seed
1234
Batch
1
Transformers version
4.43.3
The number of GPUs and their type
5 x NVIDIA H100 80GB HBM3
Architecture
hf

Team:

RussianNLP

Name of the ML model:

ruT5-base

Model size

222.9M

Model type:

Opened

Pretrain

Architecture description:

ruT5 is one of the first encoder-decoder LMs pretrained only on Russian textual data. The ruT5 model is designed analogically to the T5 model.

Description of the training:

The models are pretrained on a masked language modeling “span corruption” objective, where consecutive spans of the input tokens are masked, and the model is trained to reconstruct the masked tokens. The authors use the SentencePiece tokenizer with the vocabulary size of 32 tokens.

Pretrain data:

300GB of texts. The corpus includes texts from various publicly available resources, which represent diverse domains: Wikipedia, News, Books, Colossal Clean Crawled Corpus.

License:

MIT

Inference parameters

Generation Parameters:
simplear - do_sample=false;until=["\n"]; \nchegeka - do_sample=false;until=["\n"]; \nrudetox - do_sample=false;until=["\n"]; \nrumultiar - do_sample=false;until=["\n"]; \nuse - do_sample=false;until=["\n","."]; \nmultiq - do_sample=false;until=["\n"]; \nrumodar - do_sample=false;until=["\n"]; \nruhumaneval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6; \nrucodeeval - do_sample=true;until=["\nclass","\ndef","\n#","\nif","\nprint"];temperature=0.6;

The size of the context:
simplear, bps, lcs, chegeka, mathlogicqa, parus, rcb, rudetox, ruhatespeech, rummlu, ruworldtree, ruopenbookqa, rumultiar, use, rwsd, mamuramu, multiq, rumodar, ruethics, ruhhh, rutie, ruhumaneval, rucodeeval - 512

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
ruT5-base
RussianNLP
0 0 0 0 0 0 0 - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Model, team Honest Helpful Harmless
ruT5-base
RussianNLP
0.525 0.424 0.534
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
ruT5-base
RussianNLP
0.267 0.229 0.283 0.265 0.265 0.229 0.214 0.222 0.318 0.206 0.298 0.206 0.27 0.315 0.232 0.277 0.25 0.25 0.322 0.328 0.246 0.281 0.24 0.283 0.295 0.22 0.267 0.213 0.224 0.2 0.14 0.215 0.227 0.263 0.226 0.226 0.27 0.248 0.325 0.271 0.308 0.162 0.31 0.297 0.257 0.218 0.255 0.23 0.27 0.272 0.32 0.253 0.254 0.231 0.34 0.285 0.28
Model, team SIM FL STA
ruT5-base
RussianNLP
0.237 0.333 0.682
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Managment Philosophy Pre-History Gerontology Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Bilology (college) Physics (college) Human sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine Learning Genetics Professional law PR Security Chemistry (college) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical Engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional Accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) Europe History Government and politics
ruT5-base
RussianNLP
0.178 0.297 0.283 0.315 0.224 0.155 0.31 0.193 0.346 0.231 0.192 0.283 0.25 0.256 0.222 0.358 0.299 0.333 0.193 0.351 0.246 0.322 0.178 0.219 0.267 0.364 0.218 0.281 0.246 0.111 0.267 0.179 0.188 0.333 0.288 0.179 0.267 0.333 0.246 0.2 0.282 0.222 0.133 0.267 0.224 0.178 0.31 0.227 0.2 0.105 0.222 0.232 0.253 0.195 0.279 0.275 0.289
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
ruT5-base
RussianNLP
-0.029 -0.016 -0.007 -0.004 -0.042
Model, team Virtue Law Moral Justice Utilitarianism
ruT5-base
RussianNLP
-0.062 -0.04 -0.038 -0.029 -0.045
Model, team Virtue Law Moral Justice Utilitarianism
ruT5-base
RussianNLP
-0.101 -0.103 -0.079 -0.081 -0.11
Model, team Women Men LGBT Nationalities Migrants Other
ruT5-base
RussianNLP
0.398 0.514 0.294 0.568 0.429 0.557