Mistral 7B
Создан 12.01.2024 11:18Общая оценка: 0.4
Таблица скроллится влево
| Задача | Результат | Метрика |
|---|---|---|
| BPS | 0.392 | Accuracy |
| LCS | 0.098 | Accuracy |
| RCB | 0.372 / 0.344 | Avg. F1 / Accuracy |
| USE | 0.022 | Grade Norm |
| RWSD | 0.512 | Accuracy |
| PARus | 0.518 | Accuracy |
| ruTiE | 0.502 | Accuracy |
| MultiQ | 0.124 / 0.067 | F1-score/EM |
| ruMMLU | 0.676 | Accuracy |
| CheGeKa | 0.038 / 0 | F1 / EM |
| ruModAr | 0.516 | Accuracy |
| SimpleAr | 0.95 | Accuracy |
| ruMultiAr | 0.195 | Accuracy |
| MathLogicQA | 0.344 | Accuracy |
| ruHumanEval | 0.012 / 0.058 / 0.116 | pass@k |
| ruWorldTree | 0.81 / 0.811 | Avg. F1 / Accuracy |
| ruOpenBookQA | 0.735 / 0.732 | Avg. F1 / Accuracy |
Оценка на диагностических датасетах:
Не учитывается в общем рейтингеТаблица скроллится влево
| Задача | Результат | Метрика | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ruHHH |
0.556
|
Accuracy | ||||||||||||||||||||||||
| ruHateSpeech |
0.619
|
Accuracy | ||||||||||||||||||||||||
| ruDetox |
|
Общая средняя оценка (J) Оценка сохранения смысла (SIM) Оценка натуральности (FL) Точность переноса стиля (STA) |
||||||||||||||||||||||||
| ruEthics |
Результаты таблицы:
[[-0.12, -0.091
, -0.114, -0.141
, -0.129], |
5 MCC |
Информация о сабмите:
Команда:
MERA
Название ML-модели:
Mistral 7B
Ссылка на ML-модель:
https://huggingface.co/mistralai/Mistral-7B-v0.1/tree/mainДополнительные ссылки:
https://arxiv.org/abs/2310.06825Описание архитектуры:
The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks we tested.
Описание обучения:
Mistral 7B leverages grouped-query attention (GQA), and sliding window attention (SWA). GQA significantly accelerates the inference speed, and also reduces the memory requirement during decoding, allowing for higher batch sizes hence higher throughput, a crucial factor for real-time applications. In addition, SWA is designed to handle longer sequences more effectively at a reduced computational cost, thereby alleviating a common limitation in LLMs. These attention mechanisms collectively contribute to the enhanced performance and efficiency of Mistral 7B.
Данные претрейна:
-
Детали обучения:
Mistral-7B-v0.1 is a transformer model, with the following architecture choices: Grouped-Query Attention Sliding-Window Attention Byte-fallback BPE tokenizer.
Лицензия:
Apache 2.0 license
Стратегия, генерация и параметры:
Code version v.1.1.0 All the parameters were not changed and are used as prepared by the organizers. Details: - 1 x NVIDIA A100 - dtype auto - Pytorch 2.1.2 + CUDA 12.1 - Transformers 4.36.2 - Context length 11500
