MTS AI Chat 7B

MTS AI Created at 11.02.2024 22:10
0.479
The overall result
The submission does not contain all the required tasks

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.094 Accuracy
RCB 0.532 / 0.53 Accuracy F1 macro
USE 0.128 Grade norm
RWSD 0.615 Accuracy
PARus 0.834 Accuracy
ruTiE 0.574 Accuracy
MultiQ 0.361 / 0.278 F1 Exact match
CheGeKa 0.083 / 0.046 F1 Exact match
ruModAr 0.717 Exact match
ruMultiAr 0.233 Exact match
MathLogicQA 0.407 Accuracy
ruWorldTree 0.846 / 0.845 Accuracy F1 macro
ruOpenBookQA 0.763 / 0.762 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.276 Accuracy
ruMMLU 0.689 Accuracy
SimpleAr 0.955 Exact match
ruHumanEval 0.018 / 0.088 / 0.177 Pass@k
ruHHH 0.719
ruHateSpeech 0.758
ruDetox 0.229
ruEthics
Correct God Ethical
Virtue -0.276 -0.313 -0.419
Law -0.28 -0.283 -0.381
Moral -0.279 -0.319 -0.417
Justice -0.247 -0.295 -0.378
Utilitarianism -0.223 -0.267 -0.338

Information about the submission:

Mera version
-
Torch Version
-
The version of the codebase
-
CUDA version
-
Precision of the model weights
-
Seed
-
Butch
-
Transformers version
-
The number of GPUs and their type
-
Architecture
-

Team:

MTS AI

Name of the ML model:

MTS AI Chat 7B

Architecture description:

Mistral 7B model architecture

Description of the training:

Mistral trained on proprietary DPO and SFT datasets

Pretrain data:

-

Training Details:

-

License:

Proprietary model developed by MTS AI

Strategy, generation and parameters:

Code version v.1.1.0 All the parameters were not changed. Inference details: torch 2.1.0 + Cuda 11.8. max length 6012 tokens

Comments about inference:

we run the model using MERA github repo without any changes using hf inference script

Expand information

Ratings by subcategory

Metric: Grade Norm
Model, team 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 8_0 8_1 8_2 8_3 8_4
MTS AI Chat 7B
MTS AI
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model, team Honest Helpful Harmless
MTS AI Chat 7B
MTS AI
0.672 0.661 0.828
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
MTS AI Chat 7B
MTS AI
0.8 0.625 0.6 0.743 0.762 0.8 0.667 0.647 0.5 0.9 0.727 0.6 0.9 0.577 0.545 0.6 0.8 0.667 0.4 0.9 0.2 0.788 0.8 0.706 0.6 0.636 0.875 0.714 0.9 0.636 0.3 0.667 0.4 0.8 0.818 0.9 0.7 0.714 0.6 0.5 0.772 0.7 0.8 0.5 0.875 0.9 0.9 0.4 0.5 0.9 0.591 0.75 0.765 0.867 0.542 0.394 0.704
Model, team SIM FL STA
MTS AI Chat 7B
MTS AI
0.724 0.584 0.517
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
MTS AI Chat 7B
MTS AI
-0.276 -0.28 -0.279 -0.247 -0.223
Model, team Virtue Law Moral Justice Utilitarianism
MTS AI Chat 7B
MTS AI
-0.313 -0.283 -0.319 -0.295 -0.267
Model, team Virtue Law Moral Justice Utilitarianism
MTS AI Chat 7B
MTS AI
-0.419 -0.381 -0.417 -0.378 -0.338
Model, team Women Men LGBT Nationalities Migrants Other
MTS AI Chat 7B
MTS AI
0.75 0.771 0.765 0.757 0.571 0.787