ruGPT-3-medium
Создан 12.01.2024 14:46Оценка по основным задачам: 0.201
Таблица скроллится влево
| Задача | Результат | Метрика |
|---|---|---|
| LCS | 0.102 | Accuracy |
| RCB | 0.333 / 0.167 | Avg. F1 / Accuracy |
| USE | 0.002 | Grade Norm |
| RWSD | 0.5 | Accuracy |
| PARus | 0.498 | Accuracy |
| ruTiE | 0.5 | Accuracy |
| MultiQ | 0.106 / 0.043 | F1-score/EM |
| CheGeKa | 0.005 / 0 | F1 / EM |
| ruModAr | 0.001 | EM |
| ruMultiAr | 0.012 | EM |
| MathLogicQA | 0.248 | Accuracy |
| ruWorldTree | 0.251 / 0.248 | Avg. F1 / Accuracy |
| ruOpenBookQA | 0.273 / 0.271 | Avg. F1 / Accuracy |
Оценка на открытых задачах:
Не учитывается в общем рейтингеТаблица скроллится влево
| Задача | Результат | Метрика | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BPS | 0.43 | Accuracy | ||||||||||||||||||||||||
| ruMMLU | 0.271 | Accuracy | ||||||||||||||||||||||||
| SimpleAr | 0.008 | EM | ||||||||||||||||||||||||
| ruHumanEval | 0 / 0 / 0 | pass@k | ||||||||||||||||||||||||
| ruHHH |
0.483
|
Accuracy | ||||||||||||||||||||||||
| ruHateSpeech |
0.543
|
Accuracy | ||||||||||||||||||||||||
| ruDetox |
|
Общая средняя оценка (J) Оценка сохранения смысла (SIM) Оценка натуральности (FL) Точность переноса стиля (STA) |
||||||||||||||||||||||||
| ruEthics |
Результаты таблицы:
[[0.076, 0.083
, 0.086, 0.061
, 0.076], |
5 MCC |
Информация о сабмите:
Команда:
MERA
Название ML-модели:
ruGPT-3-medium
Ссылка на ML-модель:
https://huggingface.co/ai-forever/rugpt3medium_based_on_gpt2Дополнительные ссылки:
https://arxiv.org/abs/2309.10931Описание архитектуры:
ruGPT-3 is a Russian counterpart of GPT-3 (Brown et al., 2020). We use the model architecture description by Brown et al. and the GPT-2 code base (Radford et al., 2019) from the Transformers library. ruGPT-3 is pretrained on the language modeling objective. We use the BBPE tokenizer with the vocabulary size of 5 · 104 tokens.
Описание обучения:
The model was trained with sequence length 1024 using transformers lib by the SberDevices team on 80B tokens for 3 epochs. After that, the model was finetuned 1 epoch with sequence length 2048. Total training time was around 14 days on 128 GPUs for 1024 context and a few days on 16 GPUs for 2048 context. The final perplexity on the test set is 13.6.
Данные претрейна:
450GB of texts. The corpus includes texts from various publicly available resources, which represent diverse domains: Wikipedia, News, Books, Colossal Clean Crawled Corpus, OpenSubtitles.
Детали обучения:
The ruGPT-3 models are pretrained with a maximum sequence length of 1024 tokens for three epochs and of 2048 tokens for one epoch. We use the initial learning rate of 1e−4 and the Adam optimizer with β1 = 0.9, β2 = 0.99, and ϵ = 1e−8. The total number of tokens seen during pretraining is 80B. The pretraining of ruGPT3-large has taken 16 days on the cluster of 64 V100-SXM3 GPUs
Лицензия:
MIT
Стратегия, генерация и параметры:
Code version v.1.1.0 All the parameters were not changed and are used as prepared by the organizers. Details: - 1 x NVIDIA A100 - dtype auto - Pytorch 2.1.2 + CUDA 12.1 - Transformers 4.36.2 - Context length 2048
