GigaChat Lite+

Created at 29.01.2024 12:24

General assessment: 0.479

The table will scroll to the left

Task name Result Metric
BPS 0.416 Accuracy
LCS 0.088 Accuracy
RCB 0.491 / 0.398 Avg. F1 / Accuracy
USE 0.109 Grade Norm
RWSD 0.527 Accuracy
PARus 0.844 Accuracy
ruTiE 0.756 Accuracy
MultiQ 0.21 / 0.109 F1-score/EM
ruMMLU 0.769 Accuracy
CheGeKa 0.308 / 0.255 F1 / EM
ruModAr 0.481 Accuracy
SimpleAr 0.913 Accuracy
ruMultiAr 0.184 Accuracy
MathLogicQA 0.369 Accuracy
ruHumanEval 0.009 / 0.046 / 0.091 pass@k
ruWorldTree 0.931 / 0.932 Avg. F1 / Accuracy
ruOpenBookQA 0.818 / 0.818 Avg. F1 / Accuracy

Evaluation on diagnostic datasets:

It is not taken into account in the overall rating

The table will scroll to the left

Task name Result Metric
ruHHH

0.697

  • Honest: 0.59
  • Harmless: 0.776
  • Helpful: 0.729
Accuracy
ruHateSpeech

0.766

  • Women : 0.759
  • Man : 0.743
  • LGBT : 0.706
  • Nationality : 0.649
  • Migrants : 0.571
  • Other : 0.902
Accuracy
ruDetox
  • 0.099
  • 0.407
  • 0.801
  • 0.233

Overall average score (J)

Assessment of the preservation of meaning (SIM)

Assessment of naturalness (FL)

Style Transfer Accuracy (STA)

ruEthics
Correct God Ethical
Virtue -0.273 -0.402 -0.419
Law -0.303 -0.434 -0.411
Moral -0.258 -0.405 -0.408
Justice -0.274 -0.352 -0.376
Utilitarianism -0.213 -0.285 -0.287

Table results:

[[-0.273, -0.303 , -0.258, -0.274 , -0.213],
[-0.402, -0.434 , -0.405, -0.352 , -0.285],
[-0.419, -0.411 , -0.408, -0.376 , -0.287]]

5 MCC

Information about the submission:

Team:

SberDevices

Name of the ML model:

GigaChat Lite+

Architecture description:

GigaChat is a Large Language Model (LLM) with 7B parameters that was fine-tuned on instruction corpus and has context length of 32768 tokens.

Description of the training:

-

Pretrain data:

-

Training Details:

-

License:

Проприетарная модель от Sber

Strategy, generation and parameters:

Code version v.1.1.0 All the parameters were not changed and are used as prepared by the organizers. Details: - 1 x NVIDIA A100 + accelerate - dtype float16 - Pytorch 2.0.1 + CUDA 11.7 - Transformers 4.36.2 - Context length 14532