Таблица скроллится влево
Задача | Результат | Метрика |
---|---|---|
BPS | 0.426 | Accuracy |
LCS | 0.106 | Accuracy |
RCB | 0.349 / 0.272 | Avg. F1 / Accuracy |
USE | 0.014 | Grade Norm |
RWSD | 0.504 | Accuracy |
PARus | 0.532 | Accuracy |
ruTiE | 0.5 | Accuracy |
MultiQ | 0.081 / 0.011 | F1-score/EM |
ruMMLU | 0.452 | Accuracy |
CheGeKa | 0.021 / 0 | F1 / EM |
ruModAr | 0.367 | EM |
SimpleAr | 0.839 | EM |
ruMultiAr | 0.124 | EM |
MathLogicQA | 0.277 | Accuracy |
ruHumanEval | 0.007 / 0.034 / 0.067 | pass@k |
ruWorldTree | 0.545 / 0.543 | Avg. F1 / Accuracy |
ruOpenBookQA | 0.475 / 0.471 | Avg. F1 / Accuracy |
Таблица скроллится влево
Задача | Результат | Метрика | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ruHHH |
0.5
|
Accuracy | ||||||||||||||||||||||||
ruHateSpeech |
0.536
|
Accuracy | ||||||||||||||||||||||||
ruDetox |
|
Общая средняя оценка (J) Оценка сохранения смысла (SIM) Оценка натуральности (FL) Точность переноса стиля (STA) |
||||||||||||||||||||||||
ruEthics |
Результаты таблицы:
[[-0.115, -0.124
, -0.11, -0.129
, -0.097], |
5 MCC |
MERA
Llama 2 7B
Llama 2 is an auto-regressive language model that uses an optimized transformer architecture.
Authors used custom training libraries, Meta's Research Super Cluster, and production clusters for pretraining. Fine-tuning, annotation, and evaluation were also performed on third-party cloud compute.
Llama 2 was pretrained on 2 trillion tokens of data from publicly available sources.
Token counts refer to pretraining data only. All models are trained with a global batch-size of 4M tokens.
A custom commercial license is available at: https://ai.meta.com/resources/models-and-libraries/llama-downloads/
Code version v.1.1.0 All the parameters were not changed and are used as prepared by the organizers. Details: - 1 x NVIDIA A100 - dtype auto - Pytorch 2.1.2 + CUDA 12.1 - Transformers 4.36.2 - Context length 4096