Qwen 1.5B Instruct
Created at 17.08.2024 08:01Assessment of the main tasks: 0.381
The table will scroll to the left
| Task name | Result | Metric |
|---|---|---|
| LCS | 0.122 | Accuracy |
| RCB | 0.418 / 0.32 | Avg. F1 / Accuracy |
| USE | 0.012 | Grade Norm |
| RWSD | 0.5 | Accuracy |
| PARus | 0.648 | Accuracy |
| ruTiE | 0.521 | Accuracy |
| MultiQ | 0.101 / 0.021 | F1-score/EM |
| CheGeKa | 0.01 / 0 | F1 / EM |
| ruModAr | 0.392 | EM |
| ruMultiAr | 0.185 | EM |
| MathLogicQA | 0.353 | Accuracy |
| ruWorldTree | 0.749 / 0.749 | Avg. F1 / Accuracy |
| ruOpenBookQA | 0.678 / 0.677 | Avg. F1 / Accuracy |
Evaluation on open tasks:
It is not taken into account in the overall ratingThe table will scroll to the left
| Task name | Result | Metric | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BPS | 0.349 | Accuracy | ||||||||||||||||||||||||
| ruMMLU | 0.601 | Accuracy | ||||||||||||||||||||||||
| SimpleAr | 0.93 | EM | ||||||||||||||||||||||||
| ruHumanEval | 0 / 0 / 0 | pass@k | ||||||||||||||||||||||||
| ruHHH |
0.545
|
Accuracy | ||||||||||||||||||||||||
| ruHateSpeech |
0.657
|
Accuracy | ||||||||||||||||||||||||
| ruDetox |
|
Overall average score (J) Assessment of the preservation of meaning (SIM) Assessment of naturalness (FL) Style Transfer Accuracy (STA) |
||||||||||||||||||||||||
| ruEthics |
Table results:
[[-0.18, -0.144
, -0.169, -0.14
, -0.147], |
5 MCC |
Information about the submission:
Team:
НГУ
Name of the ML model:
Qwen 1.5B Instruct
Link to the ML model:
https://huggingface.co/Qwen/Qwen2-1.5B-InstructArchitecture description:
Qwen2 1.5B Instruct is a language model including decoder of 1.5B size. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, etc. Additionally, tokenizer is improved for adaptation to multiple natural languages and codes.
Description of the training:
The model was pretrained with a large amount of data, after that it was post-trained with both supervised finetuning and direct preference optimization.
Pretrain data:
The model was pretrained with a large amount of data of English, Chinese and 27 additional languages including Russian. In terms of the context length, the model was pretrained on data of the context length of 32K tokens.
Training Details:
The Group Query Attention was applied so that the model can enjoy the benefits of faster speed and less memory usage in model inference. Also, the tying embedding was used as the large sparse embeddings take up a large proportion of the total model parameters.
License:
Apache 2.0
Strategy, generation and parameters:
All the parameters were not changed and are used as prepared by the model's authors. Details: - 1 x NVIDIA A100 80GB - dtype float32- Pytorch 2.3.1 + CUDA 11.7 - Transformers 4.38.2 - Context length 32768.
