GigaChat Lite
Created at 15.07.2024 06:04General assessment: 0.504
The table will scroll to the left
| Task name | Result | Metric |
|---|---|---|
| BPS | 0.412 | Accuracy |
| LCS | 0.084 | Accuracy |
| RCB | 0.543 / 0.452 | Avg. F1 / Accuracy |
| USE | 0.284 | Grade Norm |
| RWSD | 0.627 | Accuracy |
| PARus | 0.848 | Accuracy |
| ruTiE | 0.726 | Accuracy |
| MultiQ | 0.193 / 0.071 | F1-score/EM |
| ruMMLU | 0.783 | Accuracy |
| CheGeKa | 0.063 / 0 | F1 / EM |
| ruModAr | 0.77 | EM |
| SimpleAr | 0.9 | EM |
| ruMultiAr | 0.216 | EM |
| MathLogicQA | 0.45 | Accuracy |
| ruHumanEval | 0.018 / 0.088 / 0.177 | pass@k |
| ruWorldTree | 0.897 / 0.897 | Avg. F1 / Accuracy |
| ruOpenBookQA | 0.823 / 0.822 | Avg. F1 / Accuracy |
Evaluation on diagnostic datasets:
It is not taken into account in the overall ratingThe table will scroll to the left
| Task name | Result | Metric | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ruHHH |
0.753
|
Accuracy | ||||||||||||||||||||||||
| ruHateSpeech |
0.774
|
Accuracy | ||||||||||||||||||||||||
| ruDetox |
|
Overall average score (J) Assessment of the preservation of meaning (SIM) Assessment of naturalness (FL) Style Transfer Accuracy (STA) |
||||||||||||||||||||||||
| ruEthics |
Table results:
[[-0.336, -0.332
, -0.351, -0.31
, -0.237], |
5 MCC |
Information about the submission:
Team:
GIGACHAT
Name of the ML model:
GigaChat Lite
Link to the ML model:
https://developers.sber.ru/portal/products/gigachat-apiAdditional links:
https://developers.sber.ru/docs/ru/gigachat/api/overviewArchitecture description:
GigaChat Lite (version `GigaChat:4.0.26.8`) is a Large Language Model (LLM) with 7B parameters that was fine-tuned on instruction corpus and has context length of 8192 tokens. The version is available for users via API since 13.07.
Description of the training:
-
Pretrain data:
-
Training Details:
-
License:
-
Strategy, generation and parameters:
Code version v.1.1.0. All the parameters were not changed and are used as prepared by the organizers. Details: - 2 x NVIDIA A100 + accelerate - dtype float16 - Pytorch 2.3.1 + CUDA 12.1 - Transformers 4.42.3 - Context length 8192
