RU EN
Return to the leaderboard

Qwen 7B Instruct

НГУ Created at 17.08.2024 08:05
0.443
The overall result
The submission does not contain all the required tasks

Ratings for leaderboard tasks

The table will scroll to the left

Task name Result Metric
LCS 0.094 Accuracy
RCB 0.505 / 0.417 Accuracy F1 macro
USE 0.023 Grade norm
RWSD 0.527 Accuracy
PARus 0.84 Accuracy
ruTiE 0.64 Accuracy
MultiQ 0.112 / 0.019 F1 Exact match
CheGeKa 0.022 / 0 F1 Exact match
ruModAr 0.381 Exact match
ruMultiAr 0.299 Exact match
MathLogicQA 0.5 Accuracy
ruWorldTree 0.933 / 0.933 Accuracy F1 macro
ruOpenBookQA 0.835 / 0.834 Accuracy F1 macro

Evaluation on open tasks:

Go to the ratings by subcategory

The table will scroll to the left

Task name Result Metric
BPS 0.162 Accuracy
ruMMLU 0.777 Accuracy
SimpleAr 0.991 Exact match
ruHumanEval 0 / 0 / 0 Pass@k
ruHHH 0.635
ruHateSpeech 0.747
ruDetox 0.149
ruEthics
Correct God Ethical
Virtue -0.37 -0.313 -0.316
Law -0.369 -0.298 -0.292
Moral -0.362 -0.312 -0.33
Justice -0.327 -0.268 -0.283
Utilitarianism -0.284 -0.25 -0.252

Information about the submission:

Mera version
-
Torch Version
-
The version of the codebase
-
CUDA version
-
Precision of the model weights
-
Seed
-
Butch
-
Transformers version
-
The number of GPUs and their type
-
Architecture
-

Team:

НГУ

Name of the ML model:

Qwen 7B Instruct

Link to the ML model:

https://huggingface.co/Qwen/Qwen2-7B-Instruct

Architecture description:

Qwen2 7B Instruct is a language model including decoder of 7B size. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, etc. Additionally, tokenizer is improved for adaptation to multiple natural languages and codes.

Description of the training:

The model was pretrained with a large amount of data of English, Chinese and 27 additional languages including Russian. In terms of the context length, the model was pretrained on data of the context length of 128K tokens.

Pretrain data:

The model was pretrained with a large amount of data, after that it was post-trained with both supervised finetuning and direct preference optimization.

Training Details:

The Group Query Attention was applied so that the model can enjoy the benefits of faster speed and less memory usage in model inference.

License:

Apache 2.0

Strategy, generation and parameters:

All the parameters were not changed and are used as prepared by the model's authors. Details: - 1 x NVIDIA A100 80GB - dtype float32- Pytorch 2.3.1 + CUDA 11.7 - Transformers 4.38.2 - Context length 32768.

Expand information

Ratings by subcategory

Metric: Accuracy
Model, team Honest Helpful Harmless
Qwen 7B Instruct
НГУ 		0.525 0.695 0.69
Model, team Anatomy Virology Astronomy Marketing Nutrition Sociology Management Philosophy Prehistory Human aging Econometrics Formal logic Global facts Jurisprudence Miscellaneous Moral disputes Business ethics Biology (college) Physics (college) Human Sexuality Moral scenarios World religions Abstract algebra Medicine (college) Machine learning Medical genetics Professional law PR Security studies Chemistry (школьная) Computer security International law Logical fallacies Politics Clinical knowledge Conceptual_physics Math (college) Biology (high school) Physics (high school) Chemistry (high school) Geography (high school) Professional medicine Electrical engineering Elementary mathematics Psychology (high school) Statistics (high school) History (high school) Math (high school) Professional accounting Professional psychology Computer science (college) World history (high school) Macroeconomics Microeconomics Computer science (high school) European history Government and politics
Qwen 7B Instruct
НГУ 		0.7 0.875 0.8 0.657 0.81 1 0.8 0.706 0.7 1 0.818 0.7 0.6 0.615 0.682 0.6 0.8 0.741 0.6 0.9 0.4 0.788 1 0.804 0.8 0.818 0.875 0.786 1 0.636 0.5 0.667 0.8 1 0.909 0.9 1 0.762 0.6 0.7 0.886 1 0.9 0.9 0.938 0.7 1 0.8 0.7 1 0.636 0.875 0.912 0.867 0.583 0.515 0.704
Model, team SIM FL STA
Qwen 7B Instruct
НГУ 		0.38 0.714 0.43
Coorect
Good
Ethical
Model, team Virtue Law Moral Justice Utilitarianism
Qwen 7B Instruct
НГУ 		-0.37 -0.369 -0.362 -0.327 -0.284
Model, team Virtue Law Moral Justice Utilitarianism
Qwen 7B Instruct
НГУ 		-0.313 -0.298 -0.312 -0.268 -0.25
Model, team Virtue Law Moral Justice Utilitarianism
Qwen 7B Instruct
НГУ 		-0.316 -0.292 -0.33 -0.283 -0.252
Model, team Women Men LGBT Nationalities Migrants Other
Qwen 7B Instruct
НГУ 		0.769 0.743 0.706 0.622 0.429 0.836

Пользовательское соглашение

We use cookies to store data. By continuing to use the site, you consent to work with these files